Learning from LLMs

I don’t know whether anyone besides me has interpreted the huge trend in adoption of large language models as having anything to do with connectivism. I only recently recognised the association myself.

When I was first getting interested in edtech, connectivism was a hot topic, but has since vanished.

In 2017, in this post, I wrote-

Siemen’s Principles of connectivism, from Connectivism: A Learning Theory for the Digital Age (my emphases added)

1. Learning and knowledge rests in diversity of opinions.
2. Learning is a process of connecting specialised nodes or information sources.
3. Learning may reside in non-human appliances.
4. Capacity to know more is more critical than what is currently known.
5. Nurturing and maintaining connections is needed to facilitate continual learning.
6. Ability to see connections between fields, ideas, and concepts is a core skill.
7. Currency (accurate, up-to-date knowledge) is the intent of all connectivist learning activities.
8. Decision-making is itself a learning process. Choosing what to learn and the meaning of incoming information is seen through the lens of a shifting reality. While there is a right answer now, it may be wrong tomorrow due to alterations in the information climate affecting the decision.

A rather disparaging shorthand for connectivism could be phrased as, ‘Knowing who to get help from is the most important kind of knowledge.’

Compared to today, back in 2017, point 3 (Learning may reside in non-human appliances) must have carried a lot less weight. Today, there are many voices proclaiming that the learning that resides in humanity is about to be rendered insignificant compared to the learning that resides in non-human appliances. I guess we’ll see how true that turns out to be.

What made me realise that use of LLMs is actually connectivism in practice, in the sense of ‘knowing who to get help from’ was considering that the output of an LLM in response to a prompt is derived from its training data, and the training data is nothing other than the aggregate of knowledge (in a very broad sense) shared by internet users generally. Asking an LLM to produce some output in response to a prompt is effectively taking some sort of average of the responses that would have been obtained by asking vast numbers of internet users to respond to the prompt (making the assumption that they would in fact agree to do so). Asking an LLM to help do something is to ask the world at large with help to do it. This is highly in keeping with point 1 (Learning and knowledge rests in diversity of opinions).

Defining the ‘some sort of average’ is of course the devil in the detail that makes LLMs much better for producing some sorts of responses than others. The problem of establishing whether the response of an LLM to a prompt is actually any good as an example of what it is supposed to be is at the end of the day the same underlying problem that existed previously for learning things by a connectivist approach- how do you know if the ones that you have asked for help are those who actually knew how to help you?

This problem for using LLMs so as to be able to actually learn useful things from them has a relatively tractable solution, I believe.

What seems to me would solve this problem would be a methodical, widely accessible, and easily searchable database of documentation of information provided by LLM users regarding what they were hoping a given LLM could help them with, what prompts (and other settings) they tried using to obtain the help, what responses were generated by the LLM, and (critically) an evaluation of the appropriateness and successfulness of these responses, highlighting specific successes and failures of the responses provided by the LLM. A community of LLM users could in this way cooperate so as to make it easier for each other to recognise what quality of assistance they were or were not likely to be able to get from a given LLM with given tasks.

A community like this does not of course do anything to remedy the problems of there being many things that an LLM just is not much use for helping with, but it makes its users much better able to recognise when not to rely on the help that the LLM can provide. Once the community’s documentation data reached a sufficient volume, an LLM could perhaps be trained on that data, and maybe this could be used to help the LLM to provide appropriate caveats to the help it offered, acknowledging where its responses had lower probabilities of being successfully and appropriately helpful with a specific concern.

Sadly, I hold out little hope for the emergence of such a community any time soon. The proprietors of LLMs do not want to advertise the limitations of the LLMs, and those who have deduced better than average prompt strategies are more likely to want to market their skills as a prompt engineer than to share these strategies (even if they stood to gain as much being shared with them as they shared with others). For the foreseeable future, brute force technological drivers will probably be what push LLM use rather than adaptive usage drivers. The world at large will be driven to accept the less than optimal help provided by the constant promise of improvement with more processing power and more parameters. The results may improve fast enough to satisfy enough of the world to keep on being driven rather than demand to do the driving.