FRESH

Hacker News

Home

Enabling Codex to Analyze Two Decades of Hacker News Data

72 points by ronfriedhaber

by zeroxfe

1 subcomments

I've done this kind of thing many times with codex and sqlite, and it works very well. It's one prompt that looks something like this:
- inspect and understand the downloaded data in directory /path/..., then come up with an sqlite data model for doing detailed analytics and ingest everything into an sqlite db in data.sqlite, and document the model in model.md.
Then you can query the database adhoc pretty easily with codex prompts (and also generate PDF graphs as needed.)
I typically use the highest reasoning level for the initial prompt, and as I get deeper into the data, continuously improve on the model, indexes, etc., and just have codex handle any data migration.

by Brajeshwar

0 subcomment

The “Hacker News - Complete Archive” on Hugging Face,[1] recently popped up here. “The data is stored as monthly Parquet files sorted by item ID, making it straightforward to query with DuckDB, load with the datasets library, or process with any tool that reads Parquet.”
Out of curiosity, I tinkered with it using Claude to see trends and patterns (I did find a few embarrassing things about me!).
1. https://huggingface.co/datasets/open-index/hacker-news

by mike_hearn

1 subcomments

I don't quite understand how Modolap differs from just asking AI to use any other OLAP engine? Both your website and the github readme just emphasise that it's idiosyncratic and your personal approach, without explaining what that is or why anyone should care.

by RockyMcNuts

0 subcomment

Could be interesting to chart quality of responses, toxicity/health of conversations, sentiment over time, impact of release of ChatGPT.
(since AI can now answer many questions that might have been topics of conversation; people can use AI to participate; people may be reluctant to participate if AI can data mine everything and link it back to them, etc. similar to Stack Overflow)

by sd9

2 subcomments

I'm kind of surprised that postgres was quite that dominated by mongodb back in the day. I remember the mongo fever, but I always thought postgres held reasonable market share. I guess it was other SQL dbs back then, I guess MySQL was still viable.

by voidUpdate

0 subcomment

When searching for references to Go, what does it actually look for? "Go" is a relatively common word, and I hardly see anyone referring to it as Golang

by hakrgrl

1 subcomments

That last chart showing the average comment length shows a clear negative downtrend, especially in recent months. I wonder why that is.

by xnorswap

2 subcomments

5% of all comments mention Claude code?
Am I reading that right?

by sam-bee

1 subcomments

Nobody who actually codes in that language ever calls it 'Golang'

by hsuduebc2

1 subcomments

I really love codex. The price/value comparison to claude code is at least from my opinion much better.

by moralestapia

1 subcomments

Do not estimate/plot DAUs/MAUs, it's not a pretty picture :'(.

by throwaway290

3 subcomments

HN data is open? Under what conditions it's distributed?

by avib99

0 subcomment

[dead]

by huflungdung

0 subcomment

[dead]

by benreesman

0 subcomment

[dead]

by cloudpeaklabs

0 subcomment

[flagged]