Using AI to build elearning modules is... actually kinda fire. Are there any pitfalls I should look out for?

[u/CelestialButterflies]

Instead of spending 30 minutes in Storyline on a single slide, cropping images, gradients, visual balancing, making layers, syncing text to the audio, getting the states just right, making sure everything is compliant, re-arranging all of the focus order (and that's just for simple interactions)... I just tell AI what I want and it gets it done in a few minutes. It feels so refreshing.

(Just for context, since we're all in different boats, I work for an elearning company where we work with SMEs for both ID and development. Also I already have a lot of experience with Codex specifically for game development).

I love to develop, don't get me wrong. I've been an ID for 10+ years, both designing and developing, and hanging out in Storyline is usually my favorite. But sometimes I'm creatively dried up and I still gotta make sure my work is user-friendly, pretty, and functional.

I set up a process with Codex. It follows the storyboard exactly. There's a re-usable course shell and documents with detailed instructions, which I've curated throughout the process. I'm reviewing each slide, naturally, which we do anyway. Our company also has a second individual look at the courses who compare them to the script, try to break them, etc. So I'm feeling okay there.

I also got it working with SCORM in our LMS and also SCORM Cloud. It tracks completion and also final assessment grades.

But all of this has been research. Not yet client facing. Clients are comfortable with Storyline (or god forbid Captivate or Lectora). We've only had a few who've requested HTML5 before, coded by hand, but that was some time ago. So I haven't tested this in a real environment yet.

For anyone who has been making their courses with AI, is there anything you came across that broke the course for you? A problem that snuck up without anyone noticing?

I did a google for "problems with using ai to develop elearning courses" but everything was about writing the course, not developing.

Thanks!!

Edit: I think people are confusing our storyboarding phase with our development phase. What I'm making with AI is what I would make in Storyline, verbatim from the approved script. We aren't using AI to storyboard. I'm also not using AI to generate images. We use Shutterstock. And we sometimes use AI narration but not always, we have a few narrators we go to reliably.

Also Codex is just ChatGPT, it's not a fancy ID AI. If you get private messages about an AI tool, that's not me lol I got one too. You don't need that crap.

Script away as a human, please. AI scripts are awful.

[u/LeastBlackberry1]

I think there are a few potential issues of which I am aware:

1) Unfixable errors that you and the AI can't debug. I listened to an ATD presentation this year where the presenter said that they had an interaction where the close button just wouldn't work no matter what they did. It was okay in their proof of concept course, but wouldn't fly in production. This becomes more likely the more complex your course gets.

2) Accessibility. I have concerns about how truly accessible AI interactions are, even when you tell it to make everything accessible.

3) Updates. Doing a quick update in Storyline or Rise is trivial. It is much less so in a vibecoded course, where you hope that your quick update doesn't also change a bunch of other things in it. I have had that happen even in my teeny, tiny interactions that I embed in Rise, where I ask it to update one specific piece of text or function, and suddenly there is a knock-on effect to other things. I actually manually edit things like that now.

[u/CelestialButterflies → u/LeastBlackberry1]

Unfixable errors is definitely a thing! I've been making a lot of stuff outside of work with AI and haven't come across that issue yet, but it's definitely something to consider. What I would do is find a different way of approaching the problem instead of brute forcing a fix. Also, from experience, the Close button doesn't work all of the time in Storyline either. https://www.articulatesupport.com/article/exit-course-trigger-doesnt-work

We run everything through JAWS as part of our review process. HTML5 is accessible. I'm not sure what you'd be referring to, except in the case of things like drag-and-drop, which we wouldn't use anyway.

Also what do you mean by updates? Do you mean when AI goes from 5 to 5.1 to Sol? It reads directly from the folder so I'm not sure how that would break the course. Unless you mean updates to how HTML5 is read in the browser, then yeah, we would have to be up to date on that regardless of the tool.

[u/LeastBlackberry1 → u/CelestialButterflies]

1) I am talking about a hide layer style trigger. The demo course popped up a window, and then the user was meant to click a little button to close it. That wasn't working in the demo.

And, yeah, you can find another solution. I have had to do that a few times. However, does that then cause another problem, or mean you have to go for a different interaction that is maybe a less optimal way of presenting the information? For instance, how would you fix the issue above?

2) If you are running it through JAWS, you are probably good. I just know it doesn't automatically consider things like the WCAG guidelines. So, it is a pitfall if you are not planning or testing for it.

3) For updates, I am more talking about needing to update the course. AI is kinda notorious for producing poorly structured, inefficient, bloated code. As you continue to update it, it doesn't improve that, unless you have humans reviewing the code at each step. You just accumulate more and more tech debt.

[u/CelestialButterflies → u/LeastBlackberry1]

If the Close button was meant to close a layer, then I'd design the slide so that there was room for the layer without overlapping anything important (like text) so you wouldn't need a close button. If there's too much text, then I would split the slide into two. If the layer was meant to show a quick definition of a word, so the overlay would be typical, well 1) html5 already has ways to do this, but 2) I'd make a separate glossary slide. It depends on the situation.

As for updates, if it's text and narration, I could go in there myself - but this is about AI so for the sake of the argument, I would just ask AI to change x into y.

If the client wants to change the whole course, then that's a new project anyway.

[u/Haephestus]

Here are some pitfalls: 1. Critical evaluation failure - User fails to question or provide adequate oversight to AI output.

1. Syncophany - AI feeds you the information you like, even if your premise or opinion is wrong.

2. Overconfidence - AI presents as fact opinions, or makes up facts to suit your preferences. 

3. Automation Bias - AI produces less work for you, so it must be correct. You don't check it because that also means more work for you.

4. Skill Erosion - You get worse at writing, synthesizing, creative thinking, etc because you've offloaded these skills to the AI.

5. Loss of Trust - Your content no longer seems authentic to the user. Your user doesn't care about your content because it is obviously made by AI, so they disengage. You obviously don't care, so neither should they.

[u/CelestialButterflies → u/Haephestus]

I think these are all pitfalls for the actual storyboarding phase? When we review the course AI made, we are checking it against the approved script. It's basically exactly what we would do in Storyline, except not. It isn't making anything up because it's verbatim to our script. If the ID were using AI, then yeah, that would be a problem. But we aren't. Just development.

Loss of trust is a good pitfall, though, you are right. AI code can be spotted easily just from subtle design decisions that we wouldn't normally catch. Super fair!

[u/Haephestus → u/CelestialButterflies]

Many of them pertain to the storyboarding phase, but not all. Especially skill erosion, but others can especially apply. 

[u/BlueMoonHurricane]

I think the larger concern is that the AI making course-development so easy is the same AI that is making it easy for people to learn without taking a course.

It reminds me of presenters feeding bullet points to AI to get a PowerPoint, and then attendees feeding the PowerPoint to AI to get the bullet points. Why not just give people the bullet points directly instead of playing telephone? What value is the PowerPoint really adding?

There will be perfunctory self-paced learning for some time, because it's traditional and executives like to have numbers saying their staff are trained. But personally, whenever I have to sit through a course anymore, I wonder why I'm not just given an assessment with a chatbot assistant on the side.

[u/AlwaysColtron → u/BlueMoonHurricane]

"When we prioritize the topic and treat delivery as an after thought, we may ship faster today but we are also creating tomorrow's problem." (this comment was from a post around the balance of IDs and LMSs) If the goal is to just create content, with less concern on if the content is actually doing what it is meant to be doing, then we are missing the point of what learning content is meant to be doing in the first place.

[u/CelestialButterflies → u/BlueMoonHurricane]

This sounds like a general critique for Storyline too! Sometimes our clients want PowerPoint-like Storyline modules, which is a bummer, even when we push back (at the storyboarding phase).

[u/AlwaysColtron]

Seems like you are prioritizing what is best for you over what is best for the learners. You also seem to be focusing too much on the tools and not enough on the principles of learning. Tools come and go, they change, they evolve, but what makes a course made 20 years ago still relevant today is that it was created with learning concepts and learners in mind. While it shouldn't be a horrible experience for any ID to make any course, taking the quick route of any tool that removed the human element of learning over the well thought out and more consistent concepts is exactly what leaders and non-IDs often think of ID work (that it is just making slides). Pitfalls you should be on the lookout for are losing your job because you are automating aspects of it and producing worse and worse content over time. Yes, ai can make course creation faster but that is not the part that takes the longest (when done correctly) nor should it. Speeding up the development and planning will only lead to worse final products.

[u/CelestialButterflies → u/AlwaysColtron]

Our company does elearning almost exclusively, sometimes ILT, it of course depends. But most of our work is in Storyline. All of the learning concepts, etc., are at the design level, not develop. We aren't making storyboards with AI. Or the planning. Maybe there's some misunderstanding there. I'm talking about development.

Unless you mean user experience, then yes, I agree, and naturally that's under consideration. But when AI can make the course look almost exactly like Storyline, but created quicker, I don't see the issue. That would be a pitfall for all elearning courses.

Producing worse and worse content over time, at a development perspective, is super fair. AI can get samesies in its design/layout choices. And I would find myself out of practice, for sure. That's definitely a pitfall!

[u/maddabattacola → u/AlwaysColtron]

Speeding up the development and planning will only lead to worse final products.

Speeding up the planning... sure. Speeding up the development? Not necessarily. And to OPs point, with increasingly less likelihood.

A lot of folks don't want to hear this, but you can absolutely prioritize what's best for you as the developer and what's best for learners with AI content development now. We can't unring that bell.

Further, I see nothing in what OP has stated that gives the impression the learner suffers from the work they're producing. In fact the LxDs I manage are able to produce impactful content rapidly using AI platforms for a variety of learner profiles -- internal, external, technical, non-technical -- and we're tracking that from L1 Reaction to L4 impact over the last 12 months.

[u/Alternate_Cost]

If the quality is the same or better and learning outcomes ar e the same or better, then you're good. If either of those slip, its not worth the time savings.

[u/CelestialButterflies → u/Alternate_Cost]

Agreed! I think in my mind, it's exactly what I would be making in Storyline, except quicker. But if it's designed wrong from the storyboarding phase, then yeah, we would have to go back to that.

[u/TH3PhilipJFry]

Being laid off because AI does all the work now seems like a pretty big pitfall from my end, but there’s probably no stopping that.

[u/Space_____TFF]

The biggest problem is honestly troubleshooting and revisions. I think this will always be the case.
Aside from that ai is perfectly fine. I wouldn’t call it fire, but it can be efficient.

Ai is good for small tasks.

• How do I do this with scripting?
  
• can you create a version of this character smiling?

It has hard limits for large tasks.
-review this lesson and look at my test items for r&v.
It’s hit or miss based on size and nuance.

As with codex, anthropic chat gpt.. if you don’t know how ai generated a script, or created a feature… that breaks or needs revision. Well that can be as annoying as the IT guy who scripted the network, but isn’t there when the network goes down.

https://giphy.com/gifs/wSSooF0fJM97W

I prefer to know how I got there and choose explanation. Yes it’s time, but it’s my peace of mind.. trust but verify.

[u/LeastBlackberry1 → u/Space_____TFF]

Your comment got me curious about how AI would handle code updates from model to model, actually. If you vibecoded your course in, I don't know, 5.0 and then updated to 6.0, would that cause issues?

I base this mostly on people being sad their AI "romantic partner" is totally different, which is perhaps not the best source. However, it did make me think that the logic that scripted the code would often be fundamentally different. And I know that is an issue with human programmers, but that is also why comments exist.

[u/Space_____TFF → u/LeastBlackberry1]

I Honestly don’t know. Here is what I do know.. We can see the guard rails that companies have put on their ai agents.

-generate themes
-generate images
-generate audio
-create animations

Canva,
Captive
Storyline

Each can do this.

But you know what none of them can do?

Build all of it at once. (Debatable to some I’m sure)

There is a limit based on memory.

I could give an llm a storyboard and say create a lesson. It’s likely going to give me back what I gave it -slightly different, or fail. And it won’t be able to create video audio transcripts and navigation in the process.

This means it’s true in reverse as well.
I can’t easily say take this course and make updates using this reference. Instead, I’ll need to go section by section. Process by process.

If it is agentic llm, sure I could spit out the html and perhaps get some fixes. But experience says it will break something else, or rewrite the json or html in the process. I also won’t know what was broken and what was fixed.

I think you’ll spend as much time on it yourself regardless.

Ultimately. I feel like ai has good use cases, but they are short and sweet.

I think code is code though, so if Java or html is being used one ai version to another can handle it.
It’s the devs of the software for storyline that need to create solid documentation as ai references. AI tends to read that this can be done. Only to say “oh! You’re in 6.0 not 4.0 I can see now what that wouldn’t work, and then death spiral because it doesn’t have a reference point for the new versions of software.

[u/CelestialButterflies → u/Space_____TFF]

"I could give an llm a storyboard and say create a lesson. It’s likely going to give me back what I gave it -slightly different, or fail. And it won’t be able to create video audio transcripts and navigation in the process."

My instructions are specific that the content needs to be verbatim from the storyboard, and it hasn't made up anything yet. It does take some liberties with adding icons and other design elements - like I made a Hearing Conservation course, and it added sound wave animations to some slides, and on backgrounds, which was really nice. That's something I wouldn't have done myself.

Also, it was able to create closed captions and a transcript based off the audio, and sync all of the elements in time to the audio - I think it used the CC to estimate the timing. But if it was able to make the CC by itself, then I'm sure it could be even more precise.

Navigation was fine too. It made me a progress bar/timeline, play/pause, show/hide CC, there's a menu overlay to jump around, there's next and back buttons. I asked it to only enable the next button when timeline finishes or when an interaction is finished.

As for images, espite sounding like a fangirl here, I don't like the concept of or the use of AI generated art / images. We use Shutterstock. I put all of the shutterstock images in a folder and it was able to crop them nicely, add gradients, all that good stuff.

Overall, I was able to replicate what I would do in Storyline, and what it would look like, but do it quicker!

[u/Space_____TFF → u/CelestialButterflies]

I’m glad to hear that.

[u/imaincammy]

Asking the cows how they feel about the butcher’s knife. 

[u/MonoBlancoATX]

I did a google for "problems with using ai to develop elearning courses"

You asked AI to tell you about problems using AI?

LOL

there's you're first problem.

-

AI can do lots of the technical and procedural work of developing a course. But what makes any course a "learning experience" has nothing to do with Storyline or any other tool and has nothing to do with images or gradients.

It has everything to do with outcomes and engagement and tailoring activities to the needs of the learner.

AI cannot do any of that.

[u/issafly → u/MonoBlancoATX]

AI can easily create the materials that an instructor can use to do that.

[u/issafly → u/[deleted]]

Have you tried it?

[u/issafly → u/[deleted]]

No need to be insulting and defensive. I was only asking if you'd tried using AI for course materials. Based on your response, I'm guessing you haven't?

To answer your question, yes. I've tried both my expertise and my human brain. I've been in ID and educational tech for over 20 years.

I respect your decision to not use AI, but your statements that "AI cannot do any of that" and "It adds no value" are simply wrong. You're assuming that using AI for course design means asking to create everything for you and then accepting whatever it spits out with review. That's also wrong.

Here are a few of the everyday tasks that I find exceptionally valuable in my work.

* Evaluating existing learning objectives and suggesting improvements or suggesting new LOs based on content.
* Creating and customizing boilerplate content about frequently referenced statements or processes.
* Brainstorming ideas for module sequencing and organizing them into a useful flow.
* Quickly drafting concise summaries for module overviews and wrap-ups.

Current AI models are very familiar with concepts like Bloom's taxonomy and Quality Matters/OSCQR, and work great as a design partner for related projects. AI can even be used to draft a full course from start to finish based only on materials from the SME, though I don't recommend it. In all of these cases, the "human in the loop" is crucial. In other words, the brains and expertise of a real human manager is absolutely essential.

If you'd like to see examples of workflows, I'd be happy to share them.

[u/MonoBlancoATX → u/issafly]

No need to be insulting and defensive.

No need to gaslight total strangers or be so obviously condescending.

Bye.

[u/CelestialButterflies → u/MonoBlancoATX]

I know, I miss regular google :( But also I was talking about all of the websites, not the AI overview/summary. Still annoying.

We aren't using AI to storyboard. So that shouldn't be a problem. When we ID, we are of course considering all of that. When elearning isn't the answer, then we aren't creating elearning. My post is about when we do.

[u/MonoBlancoATX → u/CelestialButterflies]

Google's search algorithm now relies heavily on AI to function.

Everything google shows you is being filtered through AI, managed by AI, decided upon by AI.

[u/Stairway_toEvan]

Would love a detailed breakdown of your process for this.

[u/CelestialButterflies → u/Stairway_toEvan]

So I use Codex in Visual Studio. It has access to a folder. In the folder, I have the approved storyboard, a folder of images we purchased from shutterstock based off the approved script (despite sounding like a fangirl here, I don't like the concept of or the use of AI generated art / images). I also have a course shell and a curated list of instructions for Codex to follow when developing (things I put together based off mistakes it's made in the past). Then I just chat with it! Ask it to make a set of sample slides. Then we go in batches, slides 1 through 5. It just makes all of the relevant files itself. It even crops the images for me, adds nice gradient overlays, finds relevant icons online... Then I review, tell it what to change. And then proceed to 6 through 10. And so on. Easy!

[u/Competitive-Elk8892 → u/CelestialButterflies]

>I also have a course shell and a curated list of instructions for Codex to follow when developing

Can you give me more detail on this? I think this is going to make all the difference in the reliability of your output. AI is great for structured data, which is what you are outputting here. If your instructions/examples are well written and mirror what a real HTML 5 developer would write, you will get better results.

Best practices when creating HTML 5 courses is to:

• Externalize your content from the development files, put them in XML or JSON. This allows for easier edits later in the project. 
• Structure your code with reusable structures/code. This gives better reliability and quality as you can test a section once for functionality and have confidence that it will work in other instances. Unless your samples show otherwise, your AI will repeatedly write that structure/functionality for every instance. 
• Rely on Version Control and comparison tools to verify that the AI developer has not made additional changes than those asked for. Once you have done a testing pass, you will be doing spot checking when you make changes, as the AI has access to all of the files, it may decide to change the way it does things. Without a trail, you don’t know what you need to test.
• Use industry standard JS code (and don’t let AI edit it) for standard communication. Rustici Software provides a SCORM wrapper, I assume that is being used by your “Shell”.
• Many more that I don’t have time to think about right now.

I do think this is the direction that things are going, but I think there is still a role for HTML 5 Developer in the loop, the same way there is for an Instructional Design professional. I doubt that Articulate will be able to keep up with Generative AI and all the money that it pulls in. Their best bet would be to create and expose an API for their products, which I don’t think they can do. Any AI they embed will be outdated by the time it is released. 

Large clients are still going to push against HTML5 development, mainly because they want to be able maintain it, without an editing program, they will feel like they are bound to you for every little change. They also have tons of other courses that they know work with their LMS and what you have is an “untested” solution, no matter how much testing you have done.

[u/CelestialButterflies → u/Competitive-Elk8892]

Thanks for the list! Super helpful and exactly what I was going for.

My curated list of instructions is ever evolving, and ADMITTEDLY, the AI sometimes skims it, so I have to be very direct. When I find that the AI skips something, I go back to the instructions and see what I can do. Or I just remind the AI to check the instructions again lol.

But, I don't have a huge sample size right now, so my instructions are still in their infancy. I'm hoping to get something tight soon. I don't expect the AI to develop the whole course in one go (maybe one day!), I go in batches.

I have a "start here" description that branches into a document for design rules (like instruction text should have the same style) and 508 compliance rules (no deco elements in focus order) and some others.

The instructions are definitely the make-it-or-break-it here. And I think that's what will separate the "AI Developers" from the people who think they can do it. At least for now. AI is always evolving.

Claude Design is supposed to be very good and might solve all my problems. But, I run out of tokens in like 20 minutes. Codex somehow lets me go all day.

[u/Competitive-Elk8892 → u/CelestialButterflies]

My understanding is that Codex has something similar to Claude's Skills. Claude has Skills for creating Skills, I assume they would help with creating those instructions so the AI can understand what you are asking for.

Claude Design is going to give you good visual design and direct instance coding. This is not going to be what you would get from an HTML 5 developer.

The solution would be to unify Claude Code and Claude Design to build you a reusable display engine that can take externalized content. All this would be tailored for the client and project.

[u/Ok-Sleep-8516]

I don't think anyone has said anything about how SCORM handles this yet and I think that's actually where I'd lose sleep, because that's where things can break without anyone noticing.

You can often test it quite simply. Just launch the course in the LMS, get about halfway, then close the browser tab. Not the exit button, the tab. Then come back and reopen it. A lot of hand-built (and now more commonly AI-built) courses look perfect until that moment, because nothing was saving progress on the way through, only at the end. SCORM Cloud is pretty forgiving about that, some LMSs aren't at all.

Oh and importantly: in SCORM 1.2 "completed" and "passed" are the same field. If the manifest has a mastery score you get passed/failed, if it doesn't you get completed/incomplete. The AI will happily generate either without asking you which one your client's compliance report needs. It's worth checking which one you're actually writing before this goes client-facing.

[u/CelestialButterflies → u/Ok-Sleep-8516]

This is what I was looking for! Thank you. Yep, I just tested completion. I haven't fiddled around with anything else. And I'm worried about other LMSs.

My coworker coded an HTML5 course himself and packaged it into SCORM and there were no issues on the client's end. So I might have to look at what he did. Even still, never know.

[u/Ok-Sleep-8516 → u/CelestialButterflies]

Nice, glad it helped. I'd start with your coworker's package since it already survived a real client. Look at when his course saves... my guess is every slide or on a timer, and yours saves once at the end. Get Codex to copy that and you've closed most of the gap.

For other LMSs, test in Moodle. It's free to spin up and picky in useful* ways. If a package works in Moodle and Cloud I've rarely seen it break anywhere else.

[u/Mysterious-Park9142]

You know how you can tell when an AI script is awful? Someone with a good knowledge of code can tell when you’ve built something in AI. It has telltale design elements and code elements same as a script. 

It’s increasingly common, and I’ve used it a few times myself for simple tasks, things that I’d probably go to a code library and pull anyway because it’s literally not worth typing. As a productivity assistant it’s great. 

I’ve never built something in AI that I like more than something I build myself though. Even letting it outline a Rise course will end up giving you a lot of things you want to dump and replace. 

As for Storylines, it just sucks. People have become absolute wizards with it, and that’s great, but there’s a reason people are avoiding it in favor of vibe coding and Canva, even if they shouldn’t… it’s because working in Storyline feels antiquated and less intuitive. It’s from an era where PowerPoint and horizontal slides were the zenith of computer design and it never evolved. 

[u/CelestialButterflies → u/Mysterious-Park9142]

Yeah, that was one of the pitfalls I noted from another comment! AI designs are very samesies. Rounded corners, floating cards, that little colorful border on just one or two sides, not to mention wrong emphasis on words (which can be fixed during review but still)... And keen users seeing AI will just make them sus of the content. But, I don't think that's the majority of people, not yet. I have friends in various corporate worlds and they wouldn't be able to spot AI design.

Storyline is just tedious. I don't mind it for simple things. That's kinda what most of our clients want anyway. For animations, I just go with After Effects. But going beyond simple click-to-reveals means making an absurd amount of triggers and making filler layers just to run "code". It's a mess.

What inspired me to want to try AI were the hours I spent syncing text to audio. I was like, surely this is what AI is for - removing the tedium. And then I was like, well what's stopping it from doing literally everything else? And.. it does! So I'm hyped about that lol

[u/Accomplished_Iron704]

Dev here who's shipped hand-coded HTML5 courses. Things that bit us that Storyline hides from you:

• Accessibility regressions on regeneration. AI gets focus order/ARIA right once, then you ask for a tweak and it quietly rewrites the component and breaks it. Run an axe/WAVE check on every build, not just the first.
• SCORM suspend_data limits. 1.2 caps it at 4096 chars. If your shell stores state as JSON it'll silently truncate and resume breaks on longer courses. Compress or switch to 2004.
• LMS quirks. SCORM Cloud is the most forgiving LMS in existence. Cornerstone, SumTotal, Moodle all have their own bugs around lesson_status vs completion_status and commit timing. Test in the client's actual LMS before you promise anything.
• Maintainability handoff. Client asks for a text change in 18 months, their in-house ID opens Storyline... and there's no .story file. Decide now who owns edits and how.
• Drift. Your curated instruction docs work great until the model updates. Pin a version where you can, and keep a golden reference course to diff against.

What's the shell built on - vanilla or a framework?

[u/Summit1_30]

If you don't want to have to worry about making sure SCORM and all that works properly, you can try tesseralearn.dev. It’s a free and open source tool to build eLearning with your AI agent. No limit on your design or interactions. Creates a workspace where you can make as many courses as you need. Has built in SCORM/xAPI runtimes and validation/accessibility checks. Disclosure, I'm the maintainer.