Applications and Implications: How to Think About the Impact of ChatGPT & AI in Organizations
Also, grading with ChatGPT is a mixed bag...
Applications & Implications: How to Think about AI and Your Organization
I’ve been doing a lot of thinking about the impact ChatGPT and AI is having and will have on organizations, with special focus on higher education (my current field) and business (my area of teaching and my previous life). There is a lot to think about and the way I’ve come to organize my analysis of it is in terms of “Applications” and “Implications”. Applications is about how the organization can deploy and use AI tools while Implications is about how AI might change the organization and its environment. Both interact with one another and are crucial to understand.
For example, below is a work-in-progress look at how a university can consider using AI as well as how AI might change the university itself and Higher Ed overall. One can also add a time factor to create a matrix that includes short and long term views.
The above example illustrates that a university needs its faculty to think about applying ChatGPT, generative AI and other AI tools in the classroom to improve their teaching and their own personal productivity, learning and creativity. At the same time, a university must consider the proper usage of these tools by students, faculty and administrators as well as ensuring faculty focus on providing the needed AI skills to students.
The policy part is especially crucial; as you can see from the headline above, Vanderbilt university had to apologize to its students for an administrator using ChatGPT to write an organization-wide email about the Michigan State University shooting.
Of course, also critical is determining academic guidance. Can students use it for brainstorming strategies for business cases or to outline an essay? Or not? Can faculty use it to develop a rough draft of a research paper or grade students’ work (see below my experiment using ChatGPT to grade)?
Faculty also need to consider what AI skills might be important to impart during a course, which of course implies they need to develop those skills themselves if they haven’t already.
Long term, a university needs to consider how they might apply AI to improve its creativity as an organization, how it will deliver education and as a means to keep its curriculum current. An example of the latter would be using a Microsoft Bing or Google Bard (once those tools are ready for prime time) to get input on what skills and knowledge might be required for each course to keep it current. Since these tools will be connected to the internet and as such, will be able to sweep up the latest ideas on what training students need, this would be a useful way for a professor to ensure their class is up to date.
Long term implications for universities would certainly include understanding how AI will change the skills needed by students over the next 3-5 years as AI disrupts the workplace. ChatGPT alone is changing how realtors, marketers and others are productive and is moving into areas such as law, medicine and programming.
Universities must also think through how increasingly powerful AI will impact Higher Ed’s business model. Equally crucial is for universities to ask important questions about AI’s affect on society, organizations, individuals, etc. Each unit, from philosophy and political science departments to the law and medical schools need to be asking questions and thinking about how AI affects its area of study and concern.
The Applications and Implications approach outlined above is one any organization can use to get its hands around thinking about AI. Combining both a “how to use AI” with “how will AI affect us and our stakeholders” with a short and long term lens is a powerful way to analyze and strategize around this important topic.
I tried grading with ChatGPT…the results were not what I’d hoped.
After watching this video of an instructor grading essays, I was pumped to try grading some of my students’ assignments as well. As any instructor knows, the best part of the job is being with students and the worst by far is grading. It is often a long and painful task that takes significant time away from thinking about how to improve the course or coach students. So the idea of using ChatGPT to help grade was very attractive.
In the video, the instructor uses ChatGPT to provide a letter grade, an assessment and even ideas for improvement, all based on a detailed rubric. Sounded pretty awesome, so it was time to try it myself. Unfortunately, the old demo disclaimer of “your results may vary” certainly applied here.
My first attempt seemed good. I got a numerical grade of 85 along with a solid analysis of the paper. Not bad, however, I wanted to improve the prompt to make the output be more personal, so I tried again with the same paper.
This time the paper was assigned a grade of 70. Hmmm…that’s not good. Same paper, different grade. Substantially different. So I tried again, tweaking the prompt to add the student’s name.
Grade given by ChatGPT? Now it was a 90. What is going on? Changing the prompt should not have affected the result that much. I decided to give it one more try with a different paper.
This time what gave me pause wasn’t the grade but the assessment. The assignment was about using course concepts to analyze a project the student had led, then summarizing what they did well and what they could have improved. Instead of critiquing the paper, ChatGPT told the student what they did wrong in the project, e.g., a “lack of proactive stakeholder management and communication.”
Perhaps I did something wrong in my methodology. I know large language models can “hallucinate” if they are engaged in too long a conversation or get confused by a prompt, so perhaps that was my issue. However, given the results, continuing the exercise seemed to look unfruitful, not to mention potentially problematic as far as helping to assign a proper grade, so in the end I did not use ChatGPT for grading. Perhaps I’ll try again in the future if ChatGPT and other tools like it become more robust and trustworthy. Just a reminder of the lesson of the meme below…
Hope you enjoyed the newsletter. Have a great week!