This isn’t “I want to believe”, this is “it would be irresponsible to not consider”.

  • 0 Posts
  • 48 Comments
Joined 1 year ago
cake
Cake day: September 3rd, 2023

help-circle

  • Try asking one to write a sentence that ends with the letter “r”, or a poem that rhymes.

    They know words as black boxen with weights attached for how likely they are to appear in certain contexts. Prediction happens by comparing the chain of these boxes leading up to the current cursor and using weights and statistics to fill in the next box.

    They don’t understand that those words are made of letters unless they have been programmed to break each word down into its component letters/syllables. None of them have been programmed to do this because that increases the already astronomical compute and training costs.

    About a decade ago I played with an LLM whose markov chain did predictions based on what letter came next instead of what word came next (pretty easy modification of the base code). It was surprisingly comparably good at putting sentences and grammar together when working at the letter-scale. It also was horribly less efficient to train (which is saying something in comparison to word-level prediction LLMs) because it needs to consider many more units (letters vs words) leading up to the current one to maintain the same coherence. If the markov chain was looking at the past 10 words, a word-level prediction has 10 boxes to factor into its calculations and trainings. If those words have an average of 5 letters, then letter-level prediction needs to consider at least 50 boxes to maintain the same awareness of context within a sentence/paragraph. This is a five-fold increase in memory footprint, and an even greater increase in compute time (since most operations are at least of linear order and sometimes more).

    That efficiency hit would allow for LLMs to understand sub-word concepts like alphabetization, rhyming, root words, etc. The expense and energy requirements aren’t worth this modest expansion of understanding.

    Adding a General Purpose Transformer just adds some plasticity to those weights and statistics beyond the markov chain example I use above.






  • I disagree with this point.

    I used the internet extensively as a minor to socialize and find friends and to be exposed to viewpoints different from those of my peers. If I only had my peers to socialize with, things would have been much worse off for me. I found kind and supportive influences as a minor that kept me away from the hate/conservatism/fascism that many of my classmates descended into. I learned about the world and gained skills that made me a more well-rounded person. I even met up in person with thousands of strangers and had a grand time.

    I see the gatekeeping of minors from internet spaces and worry about the impact that would have had on me and my development as a young person. If I hadn’t been welcomed as a minor online, I would not have been welcomed anywhere.

    That said, I stayed the hell away from corporate spyware like facebook and twitter that only serve to reinforce existing problematic systems, expose people to the toxic IRL social environments that they may otherwise be trying to escape, and amplify the kind of hatred and bigotry that I personally was evading.

    I miss the old internet where kids were safe. I don’t think that the solution is to ban kids; the solution is to ban platforms and profiteering incentive structures that create unsafe environments. The kids are the canaries in a coal mine. If the canary isn’t doing well, you don’t just ban it and keep digging: you get the hell out and find somewhere else to be.





  • I soak them overnight, then in the morning I drain, mix with some bouillon and water, and cook in a pressure cooker for 20 min. Then I add those beans to my daily meals in various ways.

    My standard/favorite is that I airfry a lot of veggies (mushrooms, peppers, broccoli, onion…), and I add these cooked beans to the basket with a few minutes left. I marinate the veggies in oil and spices and before they go in, and the beans are good at soaking up the leftovers stuck to the marination bowl while the veggies cook. Then I put the beans and veggies in pasta with cheese (pasta fagioli mac and cheese!), on toast with tomato and cheese (like a pizza!), baked into a casserole of some sort (often with pierogies!), with shredded potato in a frying pan (latkes!), or with rice in a tortilla (burrito!). Varying the bean type and accompanying veggies prevents stagnation.

    Sometimes I end up with extra bean broth from the pressure cooker, and I turn that into a gravy to go on french fries with some cheese and the beans and veggies to make poutine. Takes extra time (two uses of the airfrier), but super delicious. I’ll probably list that as my favorite.