Do machines hallucinate insecure code? In the blink of an eye we jumped on the AI bandwagon and pivoted from AI skepticism to AI adoption, but what did we trade off exactly? Writing secure code is tougher than it seems and we humans are getting it wrong time and time again. Even highly popular open-source software projects are repeatedly found vulnerable. So how does ChatGPT or GitHub Copilot live up to standards of secure software? developers have already embraced AI for augmented software development but let's challenge those AI tools you've come to rely on day-to-day and see how capable they are in producing secure software.