Wednesday, February 28, 2024
Tuesday, February 27, 2024
Show HN: Leaping – Open-source debugging with LLMs https://ift.tt/bIPc28l
Show HN: Leaping – Open-source debugging with LLMs Show HN: Leaping - Open-source debugging with LLMs Hi HN! We’re Adrien and Kanav. We met at our previous job, where we spent about a third of our life combating a constant firehose of bugs. In the hope of reducing this pain for others in the future, we’re working on automating debugging. We started by capturing information from running applications to then ‘replay’ relevant sessions later. Our approach for Python involved extensive monkey patching: we’d use OpenTelemetry-style instrumentation to hook into the request/response lifecycle, and capture anything non-deterministic (random, time, database/third-party API calls, etc.). We would then run your code again, mocking out the non-determinism with the captured values from production, which would let you fix production bugs with the local debugger experience. You might recognize this as a variant of omniscient debugging. We think it was a nifty idea, but we couldn’t get past the performance overhead/security concerns. Approaching the problem differently, we thought - could we not just grab a stack trace and sort of “figure it out” from there? Whether that’s possible in the general case is up for debate – but we think that eventually, yes. The argument goes as follows: developers can solve bugs not because they are particularly clever or experienced (though it helps), but rather because they are willing to spend enough time coming up with increasingly informed hypotheses (“was the variable set incorrectly inside of this function?”) that they can test out in tight feedback loops (“let me print out the variable before and after the function call”). We wondered: with the proper context and guidance, why couldn’t an LLM do the same? Over the last few weeks, we’ve been working on an approach that emulates the failing test approach to debugging, where you first reproduce the error in a failing test, then fix the source code, and finally run the test again to make sure it passes. Concretely, we take a stack trace, and start by simply re-running the function that failed. We then report the result back to the LLM, add relevant source code to the context window (with Tree-sitter and LSP), and prompt the AI for a code change that will get us closer to reproducing the bug. We apply those changes, re-run the script, and keep looping until we get the same bug as the original stack trace. Then the LLM formulates a root cause, generates a fix, we run the code again - and if the bug goes away, we call it a day. We’re also looking into letting the LLM interact with a pdb shell, as well as implementing RAG for better context fetching. One thing that excites us about generating a functioning test case with a step-by-step explanation for the fix is that results are somewhat grounded in reality, making hallucinations/confabulations less likely. Here’s a 50 second demo of how this approach fares on a (perhaps contrived) error: https://ift.tt/2fRKJXY We’re working on releasing a self-hosted Python version in the next few weeks on our GitHub repo: https://ift.tt/RDrp8VB (right now it’s just the demo source code). This is just the first step towards a larger goal, so we’d love to hear any and all feedback/questions, or feel free to shoot me an email at adrien@leaping.io! February 27, 2024 at 10:59PM
Show HN: Scribbler - An open source notebook tool for JavaScript https://ift.tt/KJSMfIi
Show HN: Scribbler - An open source notebook tool for JavaScript Scribbler is a tool to do experimentation in JavaScript using a notebook kind of environment. It runs in the browser without the need for a backend. It is deal for learning and experimentation in JavaScript. USP of Scribbler are: no login, no node/npm, can load ES 6 modules. Check the website at: https://scribbler.live . The web-app is available at: https://ift.tt/4PIvmEx . Scribbler has been built to satisfy a need for doing experimentation. Jupyter Notebook is very popular amongst python developers and data scientists for experimentation. It gives a simple interface for experimenting in python for testing code or for experimental analysis. Jupyter Notebook provides this application by running what is known as a “kernel” in the backend and giving back the results to the ui for display. It is an open source and free to use tool. Thus it has become extremely popular. As it is in Python, it requires installation of python environment and the libraries to use the tool. There are fully hosted alternatives like Google Colabs, where one can experiment in Python without installing anything. There is no similar open source tools for Javascript. There are online tools like jsfiddle/codepen etc but none that can be downloaded and used as a free tool or embedded on other solutions. Pure Javascript and JS libraries can ideally run without the need for a backend code like node.js or Python. Javascript is built to run by default in the browser. Optimization of the browser tech by Chromium (i.e. V8) and Firefox has ensured Javascript in the browser is fast and efficient. Thus we can build a good notebook tool using just front end technologies. I’ve been looking for such a solution for quite sometime mainly to test out some of the open source JS libraries and also for building some new open source projects. As I couldn’t find any solution I have built a simple tool to run javascript in notebooks. I call it Scribbler (so much for creativity). It is available as an open source solutions — free to use and modify. The source code is available at: https://ift.tt/JmBNrQK It does not require any backend technology. Users can download and use it in the file system or host it in webserver to use it on the internet. I have used Github Pages to host it. As it does not require backend, I need not buy/host a server to do that (ain’t it beautiful?). JavaScript can be used for a variety of experimentation. I’ve learnt a lot about Dynamic systems while using Scribbler to do simulations. It has also helped me in understanding some concepts of decentralized finance. I’ve also used Scribbler to solve some equations using numeric methods. Given the dynamic nature of JavaScript and its close integration with the UI, one can use it for building charting/dahsboarding tools. Scribbler can infact be even used for data science and machine learning. JavaScript has a vibrant community with a wide range of libraries available. Thus the usecases of Scribbler are limitless. I hope as more people start using Scribbler, there will be more and more applications including interactuve data science, Generative AI, scientific simulations, financial/economic applications, decentralized computing etc. Happy experimenting!! https://ift.tt/4PIvmEx February 27, 2024 at 08:47PM
Show HN: I built a tool to help you search for CivitAI's models by art style https://ift.tt/bDnI3Rj
Show HN: I built a tool to help you search for CivitAI's models by art style https://ift.tt/dfLGBOw February 27, 2024 at 11:08AM
Subscribe to:
Posts (Atom)