Through the onboarding process, we realized there is an opportunity to release our developer environment setup as a guide for the community.

LunaSec Community

Yes, we have it all locally. But we’re probably going to move it offsite at some point

<@U03DFC1FQD9> Chris (<@U02C69VKPML>) ported your code to Golang so that we could run it in our backend. We're getting the replication running right now. :pray::skin-tone-2:  <https://github.com/lunasec-io/lunasec/pull/930>

How would you prefer to be credited for that? We can obviously cite you in the source code, but is there anywhere else?

Also, I’ve found that the change listener is somewhat unsustainable as-is for us, because it has this dumb n^2 growth problem

yeah, man, this changes endpoint is wild lol

I.e.:
Change 0: Package P has versions V1
Change 1: Package P has versions V1, V2
Change 2: Package P has versions V1, V2, V3

That’s what you receive via the endpoint

So if you store it all it’s this bad quadratic growth

What I’m hacking on now is computing and storing a diff for each change

Idk how else to credit me other than mentioning me in the source code

yeah ill link to your implementation in the code

it was really well written dude, we aren't replicating the blobs, but your architecture looks really well thought out

i thought this part of the code was funny

<https://github.com/donald-pinckney/npm-follower/blob/a35709a641db2284e6e1863fcbc0b5d62657d8ec/postgres_db/src/change_log.rs#L62>

where you wrote a state machine to filter out null sequences

i ended up doing that on ingestion so i didnt have to serialize, filter, and then re-deserialize

Ah yeah that was some wild bs with Postgres’s json data type 

<https://github.com/lunasec-io/lunasec/blob/448fd165e0d9fccc75af7608fca8fec5f5c6144b/lunatrace/bsl/ingest-worker/pkg/metadata/replicator/npm.go#L167>

not as efficient as your state machine lol

i am totally just ignoring the docs that are &gt; 30mb  lol

<https://github.com/lunasec-io/lunasec/blob/448fd165e0d9fccc75af7608fca8fec5f5c6144b/lunatrace/bsl/ingest-worker/pkg/metadata/replicator/npm.go#L143>

I'm convinced that they are just trolling lol.

Did you consider using the "all docs" endpoint in Couch? It dumps everything at you in one shot with only the latest revisions of each package.

We were debating if that might be better and then just listening to the changes endpoint for updates. Potentially even throwing away old revisions of packages to only keep the latest.

We ended up just using your code as a reference because you're a smart dude and we could likely trust that your approach was the best one you tried + tested lol.

I tried natively replicating CouchDB and it did work initially but... it's painful enough from an Ops perspective that just shoving everything into Postgres is simpler (we're using AWS Aurora Serverless so we don't have to deal with administrating a CouchDB cluster).

I looked at the all docs endpoint a little bit

I `curl`ed it and it just kept downloading for a few hours, so i got tired and killed it. Seemed like the changes thing was simpler, though a bit wacky

(I'm going to move this chat to General because that's where we have been talking about Text4Shell. This chat is just about how to scrape NPM.)