I HAVE TRAPPED YOU HERE FOR 0 SECONDS
Arkansas State University
Degree comp sci
You can click on this site trust me
LANGUAGES: JavaScript, C++, Rust, Java, Python
TOOLS: Git, Node/Bun, Firebase, MySQL, Pytorch/ML, Cloud, React/Svelte
I wrote AI from scratch
EMAIL: trevorblythe82@gmail.com
PHONE: ask me
LOCATION: not here
github.com/TrevorBlythe
linkedin.com/in/trevor-blythe-31725b225/
The term "imaginary" means nothing to what it actually is.
Complex number, is a far better name.
I wouldnt have given it a name. If I needed a number to represent the phase/rotation of something, I would just say that. A complex number is a 2D vector (an arrow) with multiplication defined as (a,b) * (c,d) = (ac−bd, ad+bc)
"i" is just complex number pointing up. (x:0, y:1)
Multiplication for complex numbers is defined as:
(a,b) * (c,d) = (ac−bd, ad+bc)
This process:
1. Scales the vector’s length, and
2. Rotates it by vector's angle.
Mathematicians only did this so they could say i * i = -1, for clout. That angers me!! Its made to sound complicated.
people should be cohesive enough to break away from this technical debt.
"Self Attention" is the part of the AI model's code that computes the meaning of text.
The AI model has an array for every english word called an "embedding". It represents the words meaning. Each number in the array, represents a specific thing about the word. Ex: is the word a fruit, how red is the word, is it a verb
Each number in the array, represents a thing about the word. ex: is the word a fruit, is the word red? (this array is really long in reality).
But a words meaning, depends on context. EX: (money bank, river bank).
"self attention" changes the current words embedding, based on the words around it, so that it contains the full meaning of the current word in context.
If we had an embedding, with ONLY the number that represents being an adjective, you could dot product this embedding, with any word, to test if its an adjective.
If an AI is given the text "river bank", it needs to know if "bank" means terrain or money building. We would need to check if any other words were terrain or money related. So the AI needs a way of knowing what context the word "bank" needs. It needs to know what information to check on the other words.
Each AI has three 2D arrays called the "query matrix" (Q), "key matrix" (K), and "value matrix" (V). These three matrices, can be used to compute a distinct way context changes meaning.
For each word the AI's processing, the query and key and value matrix is multiplied by the embedding, producing Q and K and V vectors (Qv, Kv, and Vv).
A words Qv, represents what context it needs. (like being terrain or money related).
A words Kv, represents what context it can give to other words. (like being terrain or money related).
A words Vv represents what to add to other words embedding (like being terrain or a building).
Every Qv and Kv, gets dot-producted together, to check if any words can give context to each other. If they can, that words Vv gets added to the word.
"river bank", "river"s Kv matches with "bank"s Qv, "rivers" Vv added to "bank"
"money bank", "money" Kv matches with "bank"s Qv, "moneys" Vv added to "bank"
In reality there are far more ways context changes meaning, so the AI will have hundreds of sets of Q, K, and V matrices. Each Q,K, and V is called a "head". And the text goes through head after head, refining the meaning more and more.