Tuesday, April 30, 2024
Show HN: I replicated Anthropic's monosemanticity research using just my MacBook https://ift.tt/5t4Li0h
Show HN: I replicated Anthropic's monosemanticity research using just my MacBook Hi everyone, I've been working on an open-source implementation of Anthropic's research on monosemanticity ("Towards Monosemanticity"). The problem Anthropic is trying to solve is that language models are hard to interpret because individual neurons can be responsible for multiple different things. The research finds that training a small autoencoder on neuron activations can result in "features" which are much easier to interpret. When I was reading the original research, I got really excited when I realized that the models they used were really small, and I could probably train them from scratch with just my M3 MBP. My models are somewhat undertrained compared to what Anthropic produced, but I think my results are still very compelling. Let me know what you think! https://ift.tt/Qwz58uV April 30, 2024 at 09:26PM
Monday, April 29, 2024
Show HN: Attorch – PyTorch's nn module written in Python using OpenAI's Triton https://ift.tt/f5MyGmT
Show HN: Attorch – PyTorch's nn module written in Python using OpenAI's Triton attorch is a subset of PyTorch's nn module, written purely in Python using OpenAI's Triton. Its goal is to be an easily hackable, self-contained, and readable collection of neural network modules whilst maintaining or improving upon the efficiency of PyTorch. In other words, it intends to be a forkable project endowed with a simple, intuitive design that can serve as an accessible starting point for those who are seeking to develop custom deep learning operations but are not satisfied with the speed of a pure PyTorch implementation and do not have the technical expertise or resources to write CUDA kernels. There already exist a number of wonderful PyTorch-like frameworks powered by Triton, but most concentrate solely on Transformers and NLP applications, whereas attorch aims to be more inclusive by also presenting a variety of layers pertaining to areas besides NLP such as computer vision. Moreover, attorch is not an inference-only package and fully supports both forward and backward passes, meaning it can be used during training as well as inference, though its performance for the latter is generally not on par with dedicated inference engines. Questions and feedback are welcome in the comments sections. https://ift.tt/rQtjiXH April 29, 2024 at 11:37PM
Sunday, April 28, 2024
Show HN: Dotenv, if it is a Unix utility https://ift.tt/d9gQCKS
Show HN: Dotenv, if it is a Unix utility I like the idea of using dotenv files, but I dislike having to use different language-specific libraries to read them. To solve this, I created a small utility that lets you prefix any command with "dotenv" to load the ".env" file. This is how I imagine dotenv would work if it had started as a UNIX utility rather than a Node.js library. https://ift.tt/Y2Ogk83 April 29, 2024 at 12:25AM
Subscribe to:
Posts (Atom)