FRESH

Hacker News

Fine-tuning an LLM to write docs like it's 1995

161 points by taubek

by v1ne

3 subcomments

The trick about documentation is depth, not prose. You need context and understanding to write documentation "like in the old days". No amount of LLM trickery will free you from that. Once you have that source material, it's easy to re-shape it into an 80's/90's/00's doc format.
Negative example: I was looking into the German manual of my Canon EOS R5 II, and it is just fluff. Hundreds of pages, full of white space, telling me about features without actually explaining what they mean. Awful automatic translations. Their manuals used to be good (looking at my EOS 6D). But these days: oh boy.

by kjellsbells

2 subcomments

There are several reasons why old docs work. First, release velocity approximated documentation velocity. If you only released once a year, your docs had time to be polished. Second, simplicity. Think of the length of the man page for ls in Seventh Edition UNIX vs today. The constraints of the time helped here in that writers needed to get their point across in one or two 72x24 screens, not two million pixels.
Since good documentation creates a consistent mental model in the reader, cultural affinity of the writer to both source (developer) and reader helps, and the old, much smaller, computer industry was able to pull that off. I sat two cubes from my doc writer and we shared the same cultural worldview with each other and our market. It's much easier to communicate in that milieu because so much can be left unsaid.
Its possible that we are entering a Golden Age of Text, where everyone realizes that they have to feed their AI with decent information in order to have any hope of it producing good answers (especially true for complex technical products and internal corporate processes). But I am not hopeful.

by vintagedave

2 subcomments

I love old-school docs, and this was a fantastic read. But, I couldn't see the three generated doc pages linked anywhere. Did I miss something?
I'd really like to see the Win2K-style docs on REST, for example.
Edit: it was right there, in bold, too. https://gist.github.com/theletterf/0b8ee1112fbd087f3141d0cad...

by wxw

0 subcomment

> Meet Bitsavers: it’s a website that collects and scans old computer manuals and brochures. It’s an incredibly valuable repository of computer history and ancient tech writing, with mirrors available everywhere.
Wonderful! Thanks for the introduction to this resource.

by paultopia

2 subcomments

Does anyone know of any good write-ups on how to carry out this sort of task, for people who are reasonably technical (i.e., know how to code) but aren’t deep in the AI world? I feel like “customize a model based on a corpus of documents” (whether that’s “fine-tuning” or “RAG”) is a thing that everyone wants to know how to do but nobody actually explains in straightforward terms. (I pay for Gemini solely for access to NotebookLM for these purposes, but it would be nice to just be able to roll my own locally.)

by axus

1 subcomments

Anthropic eponymized Claude Shannon as the world's most powerful AI. Fabrizio the blogger named "a 7B model, trained on 1990s documentation" after Fabrice Bellard . Or perhaps his own name :)

by fga_qwrh

0 subcomment

Fine tune your mind instead. LLMs have no concept of prioritizing and cutting down information.
LLMs work for half page answers of targeted questions. All longer prose is like swimming through molasses.

0 subcomment

by badsectoracula

2 subcomments

The information about finetuning is interesting (it is something i'd like to do myself at some point, though i'll wait until i can do it with local hardware :-P). However FWIW LLMs are generally good at following a specific style when given examples.

As an example i asked Devstral Small 2 to write some docs for my LIL scripting language in the following style (this is copied from the DirectDraw documentation, edited to be text friendly):

    IDirectDraw7::CreateClipper
    ---------------------------

    The IDirectDraw7::CreateClipper method creates a DirectDrawClipper object. 

        HRESULT CreateClipper(
        DWORD dwFlags,
        LPDIRECTDRAWCLIPPER FAR *lplpDDClipper,
        IUnknown FAR *pUnkOuter
        );

    Parameters

    * dwFlags - Currently not used and must be set to 0. 
    
    * lplpDDClipper - Address of a variable to be set to a valid
        IDirectDrawClipper interface pointer if the call succeeds.
        
    * pUnkOuter - Allows for future compatibility with COM aggregation features.
        Presently, however, this method returns an error if this parameter is
        anything but NULL. 
        
    Return Values

    If the method succeeds, the return value is DD_OK.

    If it fails, the method can return one of the following error values: 

    * DDERR_INVALIDOBJECT
    * DDERR_INVALIDPARAMS
    * DDERR_NOCOOPERATIVELEVELSET
    * DDERR_OUTOFMEMORY

    Remarks

    The DirectDrawClipper object can be attached to a DirectDrawSurface and used
    during IDirectDrawSurface7::Blt, IDirectDrawSurface7::BltBatch, and
    IDirectDrawSurface7::UpdateOverlay operations.

    To create a DirectDrawClipper object that is not owned by a specific
    DirectDraw object, use the DirectDrawCreateClipper function.

    Requirements 

    Windows NT/2000: Requires Windows 2000.
    Windows 95/98: Requires Windows 98.
    Header: Declared in ddraw.h. 

    See Also

    IDirectDrawSurface7::GetClipper, IDirectDrawSurface7::SetClipper

And it did a fine job. I put the full transcript in[0] to check out. The neat bit is that it can even handle weird formats like a custom documentation format i have (which only exists in my PC because i haven't released it anywhere) for a "master document" that can then be converted to various other file types. I gave it an example of some code in that and asked it to convert the documentation to it (this is part of the transcript at the end). Then i copy/pasted the generated code to a new file (adding a few extra lines the doc system expects which weren't part of the example - BTW i did not had to modify the generated code at all) and from that i generated a CHM file[1]. FWIW here is a comparison with the DirectX page i copied[2] (though consider that the generated pages went through the doc format which forces its own style and the textual output in the transcript matches the given style better).

[0] https://app.filen.io/#/d/9f4c1225-3527-4f16-a522-0678342120c...

[1] http://runtimeterror.com/pages/iv/images/45f8df428afe4fe6b6a...

[2] http://runtimeterror.com/pages/iv/images/ee58032790a049d7e74...

by mock-possum

2 subcomments

> we’re not there yet, in part because of how much more powerful connected frontier models are
Is that why though? You need a beast of a machine to run a functional local model in my experience.
I think the big part is there’s significant sticker shock to buying capable hardware.
That said,
> weekend. I chose to try fine-tuning on two models, Llama 3.1 8B Instruct and Qwen 2.5 7B Instruct. At their size (around 8B) they run comfortably on a MacBook Air
Perhaps I spoke too soon?
Anyway
> I chose the Microsoft collection as the source of training materials. The collection contains out-of-print docs published between 1977 and 2005: more than 37 million words, covering old systems and SDKs
this strikes me as a very specific brand of 1995’s prose, spanning about 30 years. It’s a cool article though, so maybe that’s a forgivably clickbaity title.

by spacebacon

3 subcomments

Now do it without the fine tuning.
https://github.com/space-bacon/SRT
The HF zool4nd3r demo may be useful

by holoduke

4 subcomments

Who is reading docs these days? It there is one thing a LLM is good at is reading docs. I never read docs anymore and I am so happy about it.

by m_m_carvalho

0 subcomment

[flagged]

0 subcomment

by DuduZhvania

0 subcomment

[flagged]

by openclawclub

0 subcomment

[flagged]

by sspoisk

0 subcomment

[flagged]

by openclawclub

0 subcomment

[dead]

by realfutureman

0 subcomment

[flagged]

by eddysir

0 subcomment

[flagged]

by krupan

0 subcomment

I thought we were working towards curing cancer and solving world hunger with AI, but I guess slightly tweaking the writing style it outputs is more fun?