by Dave Wyer
Background
Ed Star is an add-in to AutoCAD that is used to automate the design and creation of construction drawings for commercial rooftop solar installations (see Glossary). One of the ways of automating its functionality is through the creation and use of “blocks”, which are predefined groups of AutoCAD entities (lines, circles, text, etc.) that have been assigned a name. EdStar utilized a block library that contained just shy of 1000 predefined blocks. These blocks represented solar panels, mounting hardware, wiring schematic symbols, standard construction details, etc. Collectively, these blocks were referred to as COMMBLOCKS.
Furthermore, all the COMMBLOCKS were tracked in a SQL Server database table. There was a record in the table for each block with fields for the block’s name, its source file location, inventory part number etc. The data in this table was used to populate one of EdStar’s primary UI components.
Chain of Events
Explore the sidebar on the left. You can add media to add color to your post. You can also change the cover image, optimize the SEO Settings, add categories or tags, and much more. Spend some time discovering each tab and learn how you can enhance your future posts.
On February 21, 2018, a standards engineer for the commercial design group, emailed me directly to inform me about an issue they were having their COMMBLOCKS functionality. In the email he stated:
We’re having issues with a legacy SolarCity piece of software, a plug in for AutoCAD for commercial PV and Energy Storage design known as EdStar. One of its features is to automatically pull ACAD blocks from our block library on the server at \\PHOTON\GROUPS\ProcessImprovement\4_Design\_CommercialBlockProject\Block_Archive every midnight. It then updates the list of blocks loaded for designers to use each day. We use this to disseminate design updates.
Sometime in the last month or so, these automatic updates from \\PHOTON\GROUPS\ProcessImprovement\4_Design\_CommercialBlockProject\Block_Archive stopped occurring and files not in that specific folder started appearing on the list uploaded to designers. Perhaps there was an issue related to the Photon server?
It is interesting to note that this coincides very closely to when Mr. Rollins was given the reins to the Rosetta pod.
Despite having received that request for support, and having that request acknowledged by the Maira Malik, and having Mr. Taylor follow up about a month later, Rosetta pod’s project manager, no work was undertaken on this issue for six full months. Why?
In late August of 2018, the issue the issue resurfaced again because I had worked previously on the development of EdStar, and because I was one of only two individuals left who had a functioning EdStar development environment, I was assigned to look into the problem.
Initially, I determined the SolarComm database in which the COMMBLOCKS table resided had been dropped, so I asked Ramya Makam (the DBA assigned to support the Rosetta pod) to please restore it, which she did. Then I found out I no longer had read permissions on the restored COMMBLOCKS table as I had in the past, so I made a request to the DBA team to get my correct access rights restored. I know I waited for more than a week to get those permissions restored. Once that had happened, I discovered on October 16th, 2018 that the COMMBLOCKS table no longer had the correct data in it. To fix that, I ran a little-known utility in EdStar’s code base that cleared the bad entries in the COMMBLOCKS table and repopulated it with good data.
After that had been done, I worked with a commercial designer named Joshua Troglin on October 17th, 2018 to confirm the issue had been fixed. Additionally, Joshua informed me that other EdStar functionality that had been broken was also working again, which greatly streamlined the process of passing bill of material (BOM) information to departments downstream of the commercial design team.
One week later, on October 25, 2108, I learned that the COMMBLOCKS functionality was broken again, just like it was before. Once more, I was assigned to investigate/solve the problem. Sure enough, the correct contents of the COMMBLOCKS table had been overwritten with junk data. So for a second time, I ran the EdStar utility to fix the problem. This time however, users immediately reported that the COMMBLOCKS functionality didn’t stay fixed. I was perplexed by this, so I started to work on figuring out when and how often the good data was being overwritten by bad data.
It turned out that the overwriting of data was occurring on a daily basis in the middle of night. This should not have been happening at all. So, on or around December 17th, 2018, I asked Ramya if she could trace where the instruction to overwrite the COMMBLOCKS table was coming from. She discovered a script was running every night at around 1:00 am. But despite trying, she was unable to determine its source.
Because there was a possibility that this was occurring deliberately (sabotage?), I relayed to Mr. Rollins what I had uncovered. This occurred sometime in early December, and if the problem was being deliberately introduced, would most definitely be considered whistleblowing. To my knowledge, Mr. Rollins did not take any investigative actions regarding this issue.
Also, on December 17th, 2018, I contacted Ryan Flores PE, who is a Standards Engineer for the commercial design team regarding the issue so he could understand what was happening. In that email, I stated:
Please keep this to yourself until we have an opportunity to talk because there’s an outside chance that this is being done on purpose to hinder your team’s functionality
This was most definitely a whistleblowing sort of statement.
On January 18th, 2019, my employment with Tesla ended. This was only a little more than a month after I first “blew the whistle” about my discovery and concerns surrounding this issue. It is not unreasonable for me to believe that I was retaliated against for doing this.
Some Final Thoughts
Like myself, Mr. Rollins had formerly been a member of the Osiris-Rex pod that developed EdStar
Mr. Rollins had access to EdStar’s source code
Mr. Rollins had high-levels of database access due to his position at Tesla
It was my observation that Mr. Rollins did not want to support the EdStar product. This is evidenced by the fact he failed to do anything whatsoever regarding the issue I described above.
Furthermore, on December 13th, 2018, Ryan Flores sent a lengthy email to me and Mr. Rollins requesting we attend a meeting regarding providing badly needed development support for EdStar. In a verbal conversation that occurred around the same time as that meeting with Mr. Rollins laughed about having the power to deny them.
EdStar product support never appeared in the Rosetta product roadmap.
Comments