Article: EUOLogy about RPG Campaigns as Books

[The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers]

Well, obviously, there is no perfect system, we can all agree on that. However, my experience with standardized tests, both in school and for advanced education (SAT twice, ACT, GRE, LSAT {with 4 practice tests using actual previous questions}, and many, many, many practice Network+ exams), I've found nearly all questions to be open to some interpretations. If the tests were graded by an independent party, not interested either in the system's marks OR the specific state's achievement, the test would be more acceptable. But currently they're graded by machine, which is about the only way to do it efficiently, and the machine doesn't have sufficient AI for interpretive mechanisms needed for this. Thus I reject the current possibility of writing such a test/process in the immediate future, yes.

To rephrase, I still don't like the idea of standardized tests being the primary measure of performance, but a distributed human grading system, like you mentioned, would make it a little more pallatable. Not enough to get on the bandwagon, but more pallatable.

[Skar]

The problem is that I can outperform most of my "peers" and always have been except in standardized testing environments, which are artificial and don't represent anything most people are ever going to have to deal with.

OK.  Give up on the idea that I'm advocating the use of stupidly designed tests.  I'm not and I've said it more than once.  Take the meaning of the words "standardized" and "test" and put them together.  I'm not talking about the stupid tests you keep describing.  I'm talking about a test that measures a student's grasp of material and problem solving ability against a standard.  And a good test that fills that purpose would take into account how long you have to sit at one time and how to deal with ADD kids.  We don't have them now, but we could.

I memorized those tables by frequent use and familiarity, not by sitting down and memorizing them, and that has rarely been an effective method for most people of memorizing anything. Therefore no, I don't think memorizing anything is an effective means of teaching even memorization. And if we're going to teach well, and they have to memorize something, it may as well be memorizing something that I'm actually going to use. Say, by using it so I can see that it has use. It's irrelevant the interest, i'm looking at the usefulness of a task.

Again, I'm not talking about taking the stupid techniques the teachers we have now are using and making them the standard.  You're arguing that teaching through rote memorization is pointless.  I agree with you.  But you also say that the ability to memorize has been useful.  So do whatever it takes to teach kids how to memorize and then test the skill.  Teachers will only innovate and become more effective at teaching if they have a vested interest in the end result.  Accountability.

The problem with that is that you hurt the students BEFORE you found out if it's acceptable. The idea is to maximize the efficacy long before any student loses out due to have a rotten teacher. I'm not saying that lesson plans have to fit certain molds of process, I'm just saying that they need to be reviewed so they hit important points.

I know for a fact that kids are not permanently damaged by having a bad teacher one year, I had quite a few of them and did just fine.  If teachers were accountable, as in having to turn out students that learned up to a certain standard, the bad ones would show up very quickly and could be dealt with.  

I think we agree. (I took lesson plan to mean the day to day outline rather than a general subject set) And there would be nothing wrong with defining a subject set that the teachers needed to cover. That would be accomplished by having a standard, which is tested by...dunadada... the Standardized Test.

and I'm still entirely unconvinced that Standardized testing of the students will achieve anything other than what Fell says: spitting out what the test writer thinks like.

See the post I wrote while you were writing yours.  

[Skar]

And, sorry to go on and on, I think many of the problems you have with current standardized testing are the result of the process rather than the concept.  

The test questions have to be multiple choice so a machine can grade them, since there are so many.  You have to sit for 4 hours straight because the tests are not designed into the system, they're an add on.  All this could be fixed by the distributed system I described.  Role of the government in education could be restricted almost entirely to coming up with good tests and seeing that the grading is organized in such a way to make it fair and possible to ask meaningful questions.

But I still think that even the archaic and badly designed system we have now is better than the total lack of teacher accountability we're headed for.

[The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers]

I guess my big complaint with what I think you're trying to say (and sorry, but many of your statements seemed to defend the format, particularly when you defended making kids sit through a long test) is that part of standardizing tests is standardizing grading. That can only be done two ways: through a computer as it is now or having one person grade all of them. Distributing the grading won't work, because it destandardizes the "standard" you want to achieve.

So we're back to either number 2 pencils or non-standardized tests.

[Skar]

I guess my big complaint with what I think you're trying to say (and sorry, but many of your statements seemed to defend the format, particularly when you defended making kids sit through a long test)

Not my intent.  Making an 8 year old sit through a long test is just silly.  Making a High Schooler sit through a long test could certainly be informative for the reasons I pointed out.  It all goes back to designing the tests intelligently.  Which they aren't, but I think they could be.  And as you said, they could never be perfect.  But anything would be better than no standard at all, which is where it sounds like you're heading.

is that part of standardizing tests is standardizing grading. That can only be done two ways: through a computer as it is now or having one person grade all of them. Distributing the grading won't work, because it destandardizes the "standard" you want to achieve.

So we're back to either number 2 pencils or non-standardized tests.

I disagree.  If that were true, the same grad student would have to grade everyone's test in college. (actually the professor would probably have to do it)  Not the way it works.

Forcing them all to be graded by the same person just transfers the "learn to think like the test-writer" paradigm to the grader.  The thing being graded should not be how well you can parrot the ideas of the test-writer/teacher (which the current system fosters)  The thing being graded should be whether you can answer the question.  

Essentially the standard would be something like this:

Read the following 1000 word excerpt and demonstrate comprehension  and communication ability by answering one of the following 5 questions about it in essay form clearly and completely.(Edit: realized essay form could be construed to mean something other than just clear and complete.  Clear and complete is more universally judgable and what is necessary in real life)

If you don't have the ability you won't be able to answer the question and if you do it will be obvious.

or

Demonstrate comprehension of the following class of math principles by correctly answering one of the following problems.

etc...

Questions like this, while forcing the student to actually have developed the ability to answer them, (it's nearly impossible to prepare for long answer questions by doing anything other than learning the material) lend themselves to pretty uniform grading when what you're looking for is comprehension and communication ability as opposed to whether they drew the "right" conclusion.  And it would be easy to provide guidelines for the humans doing the grading while creating the questions.

If you leave it up to the teachers and administrators we have now to determine whether their own students are "educated" everyone will pass with flying colors and the federal money will flow like water.

So how do you ensure that the students the teachers are passing are actually learning the material?

[Peter Ahlstrom]

Hmm...it seems like you are in favor of a system of testing which does not currently exist, yet you are upset with the teachers who are against the current system of testing.

[Skar]

Yes.  I am in favor of a system of testing that does not currently exist but could be called "standardized testing"  Thank you for putting it so clearly, I was failing...miserably.

And yes I am upset with the teachers who are against the system that currently exists.  My reasoning is as follows:  The teachers who speak against standardized testing offer no solution, they just want to do away with it.  That would be worse than the ganked up system we have now.  I'm afraid I don't trust the teachers in the system today to hold themselves to any standard at all.

And I've heard no alternative to "testing that measures students against a standard" offered by anyone on this board or in the system at large.

It seems obvious to me that the problems inherent in the current system are easily soluble if we can get away from the knee jerk reaction against "testing that measures students against a standard" otherwise known as "standardized testing" practiced by people who hate the current system that bears the name and teachers who know they suck and would lose their job if they became accountable and away from the federal government's need to micromanage the educational system.

If the federal government focused its money on creating and administering intelligent standardized testing it would do more to fix the system than anything it's trying to do now.

I mean for heaven's sake, the highest literacy rate this country ever had was in the beginning of the public school system when the "teaching tools" were a chalkboard, two or three books, and a desk.  Now all that money gets spent on football equipment and diversity training.  If we taught kids how to think BEFORE we tried to teach them how to use a computer or why it's not OK to tease Jimmy about the color of his skin everything else would fall into place.

[Skar]

So, it's late and I realize I've not been very clear to this point.

Forgive me if I offended, it wasn't intentional.

The question I've been ranting on for so long reduces to this:

If you don't want to test to a standard, what mechanism do you want to use to insure that students are being taught what they need to know?

[42]

telepathy

[Peter Ahlstrom]

Skar, I think you raise a good point. There should be some way to ensure that kids are learning what they are supposed to be.

I don't have any suggestions for carrying that out. I do know that I have always been very good at taking tests and that I've used that ability to as much advantage as my innate timewastering allowed, but that I've recognized several times that I scored higher than someone else who I knew actually learned the subject better or more practically applicably than me, thus indicating that all I'd learned was how to score well on the test. So at least for that reason I'm quite leery of the present system.

Yet I think it likely that the people who are out there designing the current and near-future standardized tests do have many of your goals and ideals in mind--they just aren't doing a very good job of it. Maybe making tests that really do what they're supposed to do is a lot harder than it seems.

Anyway, at the moment I do not feel strongly about the issue. I'm sure I will feel more strongly about it when I've got kids going into school. But personally I am much more afraid right now of my own kids being taught immorality in schools (from both curriculum and peers) than of them not being taught the academics (mostly for genetic reasons).

[The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers]

I disagree.  If that were true, the same grad student would have to grade everyone's test in college. (actually the professor would probably have to do it)  Not the way it works.

Well, no. Grad student examinations are rarely standardized, even in the same class.

Whenever you have a question that is not "C is the correct answer" you've got grading based on interpretation. If that interpreation is not done in exactly the same way, you are losing standardization. That's the main flaw with your proposal. You want to test against a standard, but you're system suggests a wide variation in interpretation of that problem.

Another problem: you can't test for two things at once. What if a student fully understands a biological process but can't write worth a crap? So when he describes it on paper for the test, it's incomprehensible. So he fails a science test and gets sent to more science training. But he doesn't need that at all. He needs writing training. Under your proposed test, this is not only possible, but, I would argue, probable. The same goes for someone who has sloppy writing. Yes, clear (or at least comprehensible) handwriting is a necessity. But if he writes out an accurate answer to a math question and the ones look like 7s sometimes or the 3s and the 2s get confused, it may look like he doesn't have a clue about his math. Yet that's not the training he needs.

Also, the federal government actually isn't allowed to control what is taught in the public school systems. The Constitution does not grant that priveledge to Congress, and it does grant any priveleges not specifically delineated to the states. So basically, I think they should butt out of making states take any sort of procedure unless they make an amendment to the Constitution.

[Fellfrosch]

Though it is not completely "standardized," I think that the AP testing system would work pretty well for what you're describing--it demands comprehension and knowledge more than simple test-taking ability (though some of that is inevitable), and includes both multiple-choice and essay sections. The essays are graded by a group that is well-trained and large enough to smooth out any personal irregularities.

On another note, one which we've kind of passed by already but which I'd still like to contribute to, I'd like to point out that I am the opposite of SE when it comes to taking bubble tests--I was born to blacken an oval. It comes insanely easy to me: for example, I was a National Merit Scholar and received 100% tuition for all four years of college thanks to a test that I took in 9th grade (the PSAT) with no studying and, in fact, no prior notice that I was even taking the test until the morning of. That's not because I'm a genius, it's because I know how to take tests and thrive in that environment. In a world that revolves around bubble sheet test scores, I would crush detritus like SE under the heel of my number 2 pencil-colored boots. Which, now that I think about it, wouldn't be that bad.

[The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers]

and then I would use my actual genius to destroy your pathetic graphite-and-wood structure and force you to live the life on an indigent. Oh wait. You majored in English. You already have that.

[Skar]

Well, no. Grad student examinations are rarely standardized, even in the same class.

So college students are not tested to a standard? Someone ought to tell the accreditation board.  I think they'd be upset to find that out.

Obviously I'm being sarcastic.  My point is, if we can get fair and adaptive grading to a standard on complex issues in college, surely we can get fair and adaptive grading to a standard on far simpler issues in K-12.  The single objection you seem to be bringing up is the idea that in order to have "standardized testing" (which you invariably seem to define as what you went through with the bubble-sheets and ADD)  you have to have multiple choice bubble-sheet tests.

You don't.  Witness college.  The only reason the current standardized testing people have MC Bubble-sheet tests is so they can use a computer to grade the tests.  If you could assume a minimum level of competence in all teachers you could use a distributed method like I suggested for the grading.  Yes, it's more subjective than an MC bubble-sheet but you'd be able to tell if the kids could perform the task/had the knowledge and that's all that matters.

Whenever you have a question that is not "C is the correct answer" you've got grading based on interpretation. If that interpreation is not done in exactly the same way, you are losing standardization. That's the main flaw with your proposal. You want to test against a standard, but you're system suggests a wide variation in interpretation of that problem.

Yes, but it's not a flaw.  Wasn't that your whole objection with the story about the question you got wrong when you were 9?  The grading wasn't flexible enough to recognize a correct answer that was different from other correct answers?  I think the disagreement here is I'm talking about grading to a standard and you're talking about the "standardized testing" you had when you were a kid.  I haven't been terribly clear with my distinction until my last couple of posts (thank you Ookla) but I am making a distinction.

Another problem: you can't test for two things at once. What if a student fully understands a biological process but can't write worth a crap? So when he describes it on paper for the test, it's incomprehensible. So he fails a science test and gets sent to more science training. But he doesn't need that at all. He needs writing training. Under your proposed test, this is not only possible, but, I would argue, probable. The same goes for someone who has sloppy writing. Yes, clear (or at least comprehensible) handwriting is a necessity. But if he writes out an accurate answer to a math question and the ones look like 7s sometimes or the 3s and the 2s get confused, it may look like he doesn't have a clue about his math. Yet that's not the training he needs.

Again you're talking about stupidity on the part of the organization doing the testing not a flaw in the concept of testing to a standard.  If the kid can't communicate clearly he's failing communication tests long before he's failing a biology test because he can't write clearly.  What you describe has no doubt happened but that's because we have english teachers too incompetent to administer anything other than MC bubble tests that ask their students what a "gerund" is.  (and they still have their jobs because their students are not tested to a standard and they are not held to one)

And as for the 7s looking like 1s, under a system where the kids are tested to a standard a kid whose 7s looked like 1s would still be in the first grade for that very reason.    It's because we have teachers who don't actually teach that kids are making it through the system without actually learning.  Testing to a standard would prevent that.

Also, the federal government actually isn't allowed to control what is taught in the public school systems. The Constitution does not grant that priveledge to Congress, and it does grant any priveleges not specifically delineated to the states. So basically, I think they should butt out of making states take any sort of procedure unless they make an amendment to the Constitution.

You make a good point here.  Unfortunately our schools, from elementary through University suckle at the federal money teat and unless they do what the feds want the money dries up and teachers and administrators lose their jobs.   There is, after all, nothing in the constitution that requires the feds to give money to the school systems either.

To be effective on the K-12 level the feds should either:

Cut off the money to everyone all at once, thus making the schools answerable to their local communities in one fell swoop.  I think that would end up answering most of my concerns.

Or

Somehow ensure that the teaching going on is effective.  I don't know so I ask again, how do they do that without testing to a standard?  

[The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers]

My point is, if we can get fair and adaptive grading to a standard on complex issues in college, surely we can get fair and adaptive grading to a standard on far simpler issues in K-12.  The single objection you seem to be bringing up is the idea that in order to have "standardized testing" (which you invariably seem to define as what you went through with the bubble-sheets and ADD)  you have to have multiple choice bubble-sheet tests.

You obviously misunderstand me, because I have NOT always defined it that way. I am saying that no, colledge tests are NOT standardized. Each class is usually held to the same general standard, but the standard that a class in the Bible is going to be significantly higher at say, Notre Dame than it will be at a community college. Yet they both offer degrees. Wonder why that is? Oh yes, it's because college isn't held to a single nationwide standard. There are minimums to be accredited, but that doesn't in any way say that it's standardized in terms of how the classes are graded and the assignments/tests scored.

If you could assume a minimum level of competence in all teachers you could use a distributed method like I suggested for the grading.

The very serious problem with this is that we're using the tests to DETERMINE the minimum level of competence of the teachers. Thus you cannot assume that there will be a minimum level of competence in the graders, because that hasn't been assessed.

You mentioned before that one year of bad teaching won't permanently damage a student. That contradicts in a way what you're saying. If that won't do any long-term harm, why are we worried about getting rid of those teachers? There are still very many very good teachers. My point is that you have to verify that the teachers are good before they go into the program. If you can verify that the teacher is doing what he should BEFORE he goes in to teach, you will do a lot more good than letting him go out there, teach poorly, then assess how well he did. It may not permanently harm, but it doesn't help either, and that, frankly, is waste of taxpaying dollars.

Yes, but it's not a flaw.  Wasn't that your whole objection with the story about the question you got wrong when you were 9?  The grading wasn't flexible enough to recognize a correct answer that was different from other correct answers?  I think the disagreement here is I'm talking about grading to a standard and you're talking about the "standardized testing" you had when you were a kid.  I haven't been terribly clear with my distinction until my last couple of posts (thank you Ookla) but I am making a distinction.

No, that's not at all what the disagreement is. I'm saying that once you insert the interpretive elements, especially ones as vast as you are proposing, you destandardize the grading, and thus the test is no longer standard for all students. The grading process is part of the larger testing process. That's why my complaint about the wrong answers before is valid: standardized tests are at their heart flawed in some ways, and are not a useful way of checking minimum competence.

Again you're talking about stupidity on the part of the organization doing the testing not a flaw in the concept of testing to a standard....And as for the 7s looking like 1s, under a system where the kids are tested to a standard a kid whose 7s looked like 1s would still be in the first grade for that very reason.    It's because we have teachers who don't actually teach that kids are making it through the system without actually learning.  Testing to a standard would prevent that.

No, that's very much a deep flaw in the system. You are proposing that a messy writer, no matter how capable of processing the concepts and understanding them and putting them to use cannot ever be moved out of the first grade. That is absolutely absurd. I've had a LOT of handwriting training. I got a lot of focus on it because I have particularly bad handwriting. It never got better. It's just not somethign I do. Yet I was able to graduate college. It would be an awful shame were I to still be in 1st grade at age 31 because my handwriting was poor. Sorry, no, that doesn't fly. The grading needs to be separated more before it's anywhere near acceptable.

Somehow ensure that the teaching going on is effective.  I don't know so I ask again, how do they do that without testing to a standard?  

I don't know either, however, I feel the case is strong enough against more standardized testing, however the test is structured, that we can determine it is not the answer, and to look somewhere else. Just because we don't know the right way to act does NOT mean that we should act wrongly.

However, while I don't know the solution, I go back to my program evaluation. Have the plans and material reviewed before the teacher goes into the school year. Have the TEACHER take tests. While I don't agree that a student will have legitimate problems with any form of standardized tests, any teacher who has those same problems isn't qualified to teach.