Analysis and evaluation of the final version of the PhD text – The most importan questions

Methodologists often raise common criticisms and inquiries when assessing PhD final texts. Please review, verify your text, and respond to all questions below. You don’t need to change your work’s structure and content; instead, you can compare its standards with the given recommendations.

Contents of the doctoral dissertation text:

1. Confirm titles, chapters, and the accuracy and precision of numerical notations.
2. Introduction and conclusion – should not be numbered.
3. Verify the content:

Introduction

I. AIM, OBJECTIVES AND RESEARCH RATIONALE

1. Research problem (veoma pecizno opisati i indentifikovati problem)
2. Research aims and objectives,
3. Research questions,
4. Research theoretical rationale,
5. Research philosophical approach and framework and
6. Research milestones, risks, and limitations.

II. LITERATURE REVIEW

1. Define the main terms, concepts, theories, practices and
2. Defining the primary and most significant concepts for the research field.
3. Selecting and explaining the model of literature review used.
4. Reviewing theoretical and practical conclusions of the literature review.

III. RESEARCH METHODOLOGY AND PRESENTATION OF RESULTS

1. Research aim, objectives, hypothesis, or problem-solving,
2. Research design (primary and secondary gathering of information, qualitative and quantitative research including at least 3 – 5 research methods or techniques,
3. Research onion and a brief description of the methods and techniques used and
4. Research results, measurement and indicators, presentation, and database analysis.
5. Do the research questions and hypothesis relate to the literature review, and in which way?

IV. FINDINGS AND ANALYSIS

1. Introduction to Findings
2. Descriptive Statistics or Data Overview
3. Analysis of Research Questions or Hypotheses
4. Interpretation of Results
5. Discussion of Findings
6. Comparison with Previous Studies
7. Implications of Findings
8. Limitations of the Study
9. Recommendations for Future Research

These subtitles are meant to provide structure and organisation to the presentation and discussion of the research findings, allowing readers to understand the significance and implications of the study’s results.

V. RECOMMENDATIONS AND OPPORTUNITIES FOR FURTHER RESEARCH

1. Recommendations for Practice and Policy
2. Suggestions for Future Research
3. Practical Implications and Policy Implications
4. Areas for Further Investigation
5. Future Research Directions
6. Research Gaps and Opportunities
7. Suggestions for Methodological Improvements
8. Potential Extensions of the Study

These main titles provide a framework for presenting suggestions and insights derived from the study’s findings, guiding future research endeavours, and advancing knowledge in the field.

CONCLUSION
PERSONAL REFLECTIONS
REFERENCES
BIBLIOGRAPHY
APPENDICES
Appendix A – The Research Questionnaire
Appendix B – Gantt chart for the PhD thesis project
Appendix C – Risk Management in the PhD thesis project

During the PhD completion, read and follow A Model for Producing Scientific Papers and Original Research at www.miodragivanovic.com/blog. Read, A model for producing scientific papers and original research will be beneficial.

OBSERVATIONS ON SPECIFIC PARTS OF THE DOCTORAL DISSERTATION TEXT AND SUGGESTIONS TO THE AUTHOR

(1) Observations on the research problem and recommendations to the author
I recommend to the author of the doctoral dissertation to consider (and answer to themselves) the following questions:

1. Is the research problem clearly defined?
2. What is the actual problem that this research is attempting to solve?
3. Does the section “Research problem” contain a “problem statement” in the sense in which that term is applied in the field of scientific research methodology?

Explanation of the recommendation: In science and scientific inquiry, the term “problem” implies recognising a gap in existing knowledge or understanding, often characterised as a deficiency within the corpus of established scientific knowledge within a specific domain. This conception views a problem as inherently linked to a question or an interrogative statement, reflecting an area where knowledge is lacking or incomplete within the current scientific framework. Accordingly, a scientific problem is construed as an inquiry that remains unanswered by the existing body of scientific knowledge.

The recent publication discusses the formulation of research problems, stating: “In fact, how to concretely state the research problem(s) in a specific research study is by itself a research problem on a different scale, of course. Should we put some interrogative statement(s) to be examined as the research problem? Or should it be a collection of affirmative propositions (which could be a single proposition in some cases) that have already found some evidential support? Or, do we start with a bunch of negative statements, putting at nothing claims or findings of some earlier research? Or, could the starting research problem contain some mix of these three types? Commonly accepted practices rather than standards or norms are sought. And such practices may reveal variations across fields of research.

In empirical research, the problem formulation must avoid any confusion about the connotations of the problem so that the research worker and the beneficiaries of the research results can match the results with the connotations to judge whether the research work has fully met the connotations. And right in this context, a clear statement about the scope of research becomes a must.” (Mukherjee Shyama Prasad (2020) A Guide to Research Methodology; An Overview of Research Problems, Tasks and Methods. New York, NY: CRC Press Taylor & Francis Group. pp.46-7)

“A research problem is an enigmatic or troubling condition. Researchers identify a research problem within a topic of interest. The purpose of research is to “solve” the problem—or to contribute to its solution—by generating relevant evidence. Researchers articulate the problem in a problem statement and explain the need for a study by developing an argument.” (Polit, D. F., & Beck, C.T. (2020). Nursing Research: Generating and Assessing Evidence for Nursing Practice (10th ed.). Wolters Kluwer.)

“A good statement of the problem explicitly identifies existing gaps in the literature. These gaps may be content areas not addressed or flaws in the research methods that have been used (e.g., all of the studies have been quantitative studies, so we have not heard the voices of participants through qualitative studies). If the discussion of the problem addresses these gaps, there is no need to repeat information; instead, focus on how the missing knowledge that is needed will add to the literature and make an important contribution.” (Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage.)

(2) Observations on research questions and recommendations to the author

I recommend that the author of the doctoral dissertation consider (and answer for themselves) the following questions:

Are the specified (operational) definitions of key variables within the formulations of research questions in this (predominantly quantitative) study?

Explanation of the recommendation:
In the formulated research questions:
“The principal research question of this study is: how has the failure to reconcile impacted the strategic goal for EU membership, peace, security, and cooperation?

1. How has the lack of trust and mutual respect affected the security doctrine of both sides?
2. How has the failure to reconcile affected both sides’ overall stability?
3. How does the lack of trust affect the perception of ordinary people’s personal and collective security? “Variable, such as: “lack of trust and mutual respect”, “failure to reconcile”, “security doctrine”, “overall stability”, and “perception of the personal and collective security of ordinary people “. In the dissertation text, they are not operationally defined (which is characteristic of quantitative research).

(3) Observations on the methodological approach and recommendations to the author
I recommend that the author of the doctoral dissertation consider (and answer for themselves) the following questions:

1. Is the specific type of survey explicitly stated in this research?
2. Why do none of the bibliographic references cited in the literature review pertain to survey methodology?
3. Is it appropriate to apply a survey in a doctoral dissertation without prior study of significant texts on survey methodology?

Explanation of the recommendation: In this version of the dissertation text, it states: “The research plan itself was divided into several phases: Phase planning and preparation of the survey sheet (determining the content and scope of the survey, determining the way to conduct the survey, determining the type of sample, determining the number of participants in the survey, determining the content of the survey sheet, uploading the survey sheet to Google Forms, as well as determining the dynamics of the research process) ….”
“The research that was conducted is considered exploratory research.” (p.69)
Premature, in my opinion, is the initiation of employing a “survey” in doctoral dissertation research without prior study of significant texts on “survey” methodology (such as, for example:
1. Callegaro, M., K. L. Manfreda & V. Vehovar (2015) Web Survey Methodology. London: SAGE Publications Inc.
2. Brenner, P.S.(Ed.)(2020) Understanding Survey Methodology: Sociological Theory and Applications. Springer.; Biemer,P., de Leeuw, E.,Eckman, S. et al (Eds.) (2017)
3. Total Survey Error in Practice. Hoboken, NJ: Wiley.; Lavrakas, P. J. (ed.) (2020) Experimental methods in survey research: techniques that combine random sampling with random assignment. N. J: Wiley. i drugi).

(4) Observations on sampling and recommendations to the author
4.1 Concerning the target population and sampling strategy

I recommend that the author of the doctoral dissertation to consider (and answer for themselves) the following questions:

1. Is the target population clearly defined and described?
2. Is there a clear description of the sample frame used to identify this population?
3. Is the sampling strategy appropriate for obtaining responses to the research questions posed in this study?

Explanation of the recommendation: “Groves adopted and built upon the growing tradition within survey research to utilise the Total Survey Error (TSE) framework to help survey researchers plan and interpret their research design. The TSE approach identifies major types of errors (bias and/or variance) inherent in survey designs including coverage error, sampling error, nonresponse error, and measurement error; see also Biemer et al. (2017) and Groves and Lyberg (2010). (Lavrakas, P. J. et al. (2020) Probability Survey-Based Experimentation and the Balancing of Internal and External Validity Concerns. In:
Experimental methods in survey research: techniques that combine random sampling with random assignment / edited by Paul J. Lavrakas, N. J: Wiley.)

“The total survey error (TSE) framework offers a way of looking at the quality of survey data (Biemer, 2010; Groves and Lyberg, 2010; Groves et al., 2009). The TSE framework takes into account both the representation dimension (i.e., coverage, sampling, nonresponse, and weighting adjustment error) and the measurement dimension (i.e., specification, measurement, and processing error) of the target population (Groves and Lyberg, 2010).” (Kappelhof, J. (2017). Survey Research and the Quality of Survey Data Among Ethnic Minorities. In: Total Survey Error in Practice. Biemer, P., de Leeuw, E., Eckman, S. et al Eds. Hoboken, NJ: Wiley.)
“Coverage error refers to error (primarily in the form of bias) that can result if one’s sampling frame does not well represent one’s target population. Coverage error can also result from within unit noncoverage associated with the respondent selection technique that is used (cf. Gaziano 2005).” (Lavrakas, P. J. et al. (2020) Probability Survey-Based Experimentation and the Balancing of Internal and External Validity Concerns. In: Experimental methods in survey research: techniques that combine random sampling with random. P. J. Lavrakas ed. N. J.: Wiley.)

4.2 Regarding the sample
I recommend that the author of the doctoral dissertation consider (and answer for themselves) the following questions:

1. How much cognitive caution (and moral responsibility) is required when drawing conclusions based on data obtained from a non-probabilistic sample (such as a “convenient sample”)?
2. Should the dissertation text include a description of the calculation of the necessary sample size or an explanation/justification for the sample size in this study?

Explanation of the recommendation: The text of this version of the dissertation states: “The research sample was convenient (opportunistic, convenient sampling or accidental sampling) of residents on the territory of Kosovo. Data collection will occur from October 2021 to July 2022 via Google Forms. A pilot study was conducted on a sample of 135 respondents to test the quality of the survey. After the analysis of the pilot research, the final version of the survey was compiled, and the research was conducted on a new group of 379 respondents from all over the territory of Kosovo.” (p.4)
“Sampling error refers to the error, i.e. uncertainty (in the form of variance), in every sample survey that occurs merely by chance because data are gathered from a sample of the population of interest rather than a full census being conducted. When a probability sample design is employed, the researcher can calculate the size of the sampling variance associated with the particular sample design.” (Lavrakas, P. J. et al. (2020) Probability Survey-Based Experimentation and the Balancing of Internal and External Validity Concerns. In: Experimental methods in survey research: techniques that combine random sampling with random assignment. P. J. Lavrakas ed. N. J.: Wiley.)
“With non-probability samples, the inclusion probabilities are unknown. Consequently, without further assumptions – which are usually risky and impossible to verify – in principle, this prevents any standard statistical inference calculations. In addition, non-probability samples often have a selection bias (Bethlehem & Biffignandi, 2012).
Nonetheless, researchers often and even routinely apply the standard statistical inference approaches with non-probability samples in exactly the same way as with probability samples, although in principle, this could not and should not be done because the basic assumption (i.e. known probabilities) is not fulfilled. When this is done without any consideration of the assumptions and with no warning, it might be denoted a problematic practice, so it seems reasonable to consider using a distinct terminology to separate it formally from the probability setting. Correspondingly, we might use the term indications instead of estimates (Baker et al., 2013).”‘ (Callegaro, M., K. L. Manfreda & V. Vehovar (2015) Web Survey Methodology. London: SAGE Publications Inc.)

(5) Observations on data collection and recommendations to the author

I recommend that the author of the doctoral dissertation consider (and answer for themselves) the following questions:

1. Is the necessary information about “nonresponse error” elements communicated in the dissertation text?
2. Is the necessary information about “measurement error” elements communicated in the dissertation text?
3. Are the items of the applied questionnaire sufficiently aligned with the research questions (and objectives) set in this study?
4. Do the collected data enable the acquisition of answers to the research questions?

Explanation of the recommendation: “Nonresponse error refers to the possible error (primarily in the form of bias) that can result if those sampled units from whom data are not gathered (e.g. not contacted or refusing households) are different in meaningful ways from those sampled units from which data are gathered (cf. Groves and Couper 1998). Nonresponse error can occur at both the unit level and the item level.” (Lavrakas, P. J. et al. (2020) Probability Survey-Based Experimentation and the Balancing of Internal and External Validity Concerns. In: Experimental methods in survey research: techniques that combine random sampling with random assignment. P. J. Lavrakas ed., N. J: Wiley.)
“Also, nonresponse may have further unobserved effects on the composition of the net sample which potentially limits generalizability. (Kunz, T. & M. Fuchs (2020) Using Experiments to Assess Interactive Feedback That Improves Response Quality in Web Surveys. In: Experimental methods in survey research: techniques that combine random sampling with random assignment. P. J. Lavrakas ed., N. J: Wiley.)

“Measurement error refers to the possible error (in the form of bias and/or variance) associated with a survey’s questionnaire, respondents, interviewers, and/or mode of data collection (cf. Biemer et al. 2004).”…. “Measurement error within the TSE typology is a much broader concept as it explicitly includes the full data collection environment encompassing the instrument (or tool) by which the data are captured – including item wording and ordering if a questionnaire is used – and extends to the role of the interviewer when one is present. It also includes the mode of data collection and the role of the respondent herself/himself in providing reliable and valid data.” (Lavrakas, P. J. et al. (2020) Probability Survey-Based Experimentation and the Balancing of Internal and External Validity Concerns. In: Experimental methods in survey research: techniques that combine random sampling with random assignment. P. J. Lavrakas ed., N. J: Wiley.)

(6) Observations on data analysis and recommendations to the author

6.1 Observations on quantitative data analysis and recommendations to the author

I recommend that the author of the doctoral dissertation consider (and answer for themselves) the following questions:
1. Is the application of procedures for quantitative data analysis explained in the dissertation text – is the connection between research questions/objectives and the applied data analysis procedures clarified?
2. Is the quantitative analysis aligned with the research questions and suitable for obtaining answers to the research questions (and achieving the research objectives)?

Explanation of the recommendation: Quantitative data analysis needed to be sufficiently directed towards the research questions. Two facts confirm this:
Firstly, the variables contained in the research questions: “lack of trust and mutual respect”,; “security doctrine of both sides”,; “failure to reconcile”,; “both sides’ overall stability”,; “lack of trust”; “perception of the personal and collective security of ordinary people” do not appear in the application of hypothesis testing procedures (Mann-Whitney U test; chi-square test; Kruskal-Wallis Test) – (which may partly be due to the absence of definitions of these variables in this version of the doctoral dissertation text);
Secondly, in the application of hypothesis testing procedures (Mann-Whitney U test; Chi-square test; Kruskal-Wallis Test), other variables such as:

In the application of the Mann-Whitney U test: “assessment of political security” (p.76), “assessment of the general economic situation in Kosovo” (p.79), “assessment of the quality of life and standard of living in Kosovo” (p.80), “importance of politics in personal life” (p.81), “trust in political leaders” (p.83), “estimation of tolerance between different communities” (p.89-90), “Evaluation of cooperation (work) with people from former enemy communities” (p.97), “actual interest in genuine reconciliation” (p.113), “the assessment of whether residents of Kosovo are nationalists” (p.116), “the assessment of the importance of EU membership as an incentive for accepting compromises” (p.133);

In the application of the Chi-square test: “reconciliation as a preventive measure to prevent new conflicts and the Employment status of the respondent” (p.109), “support for European integration” (p.122), “acceptance of painful compromise as a sacrifice for lasting peace” (p.130), “the attitude that it is better to invest money in economic development instead of military investments” (p.141), “the attitude that Kosovo should become a member of NATO” (p.145; p.146; p.147; p.148), “the most trusted attitude” (p.149; p.150);
In the application of the Kruskal-Wallis test: “The assessment of political security in Kosovo” (p.77; p.81), “The importance of politics in personal life in Kosovo” (p.82), “Trust in political leaders” (p.84), “estimates of the chances of emigration” (p.85), “trust between different communities” (p.87; p.88), “the assessment of tolerance” (p.90; p.94), “cooperation (work) with people from former enemy communities” (p.97; 98), “the current political and security situation” (p.105), “the real interest in true reconciliation” (p.113), “the assessment of whether Kosovo residents are nationalists” (p.116), “the impact of nationalism on political decision-making” (p.118), “the assessment of the importance of EU integration” (p.126), “the assessment of the sense of security with increased investment in the military” ( p.139), “the assessment of the feeling of security if the wax investment came from Serbia” (p.140).

6.2 Observations on qualitative data analysis and recommendations to the author

I recommend that the author of the doctoral dissertation consider (and answer for themselves) the following questions:

1. Was the applied qualitative analysis directed towards the research questions and suitable for obtaining answers to the research questions and achieving the research objectives?
2. Should the dissertation text have described how “framework analysis” was conducted in this study?
3. Should the dissertation text have described in an appropriate section how the qualitative data collected in this study were coded (indexed) in “framework analysis”?
4. Should the dissertation text have shown how the data were visually represented in “framework analysis” (e.g., maps, matrices)?

Explanation of the recommendation: In the section “Research aims and objectives” of this version of the dissertation text, it states: “This broadening of guiding principles defining national security goals enables the inclusion of reconciliation as a national security goal. This is going to be achieved by using framework analysis as a method of analysis for the collected qualitative data, both from previous studies and the current study. The foremost concern of framework analysis is to describe what occurred in a specific setting, thereby allowing researchers to draw logical conclusions based on the data (Delve, 2020). Framework analysis provides a more step-by-step approach and is primarily used for applied research, as in this study.”
In the text by the authors who developed the “framework analysis” method, it states: “The name ‘Framework’ comes from the ‘thematic framework’ which is the central component of the method. The thematic framework is used to classify and organise data according to key themes, concepts, and emergent categories. As such, each study has a distinct thematic framework comprising a series of main themes subdivided by a succession of related subtopics. These evolve and are refined through familiarisation with the raw data and cross-sectional labelling. Once it is judged to be comprehensive, each main theme is displayed or ‘charted’ in its matrix, where every respondent is allocated a row, and each column denotes a separate subtopic. Data from each case is then synthesised within the appropriate part(s) of the thematic framework.” (Ritchie, J. & Lewis, J. (eds) (2003) Qualitative Research Practice: A Guide for Social Science Students and Researchers. London: Sage. p.220).

In this version of the dissertation text, the method of conducting “framework analysis” in this study is not clearly described (nor are the results obtained by applying “framework analysis” provided).

(7) Observations on Results and Conclusions and Recommendations to the Author

7.1 On Results

I recommend that the author of the doctoral dissertation consider (and address) the following questions:

1. Is the interpretation of the results sufficiently supported by data?
2. Do the results directly relate to the research questions?
3. Are the results summarised in relation to the research questions in this version of the dissertation text?
4. Do the results relate to the research objectives?
5. Are the results summarised in relation to the research objectives in this version of the dissertation text?
6. Is there explicit discussion in this version of the dissertation text about the generalisation (“external validity”) of the research results?
7. Is there consistency between the research questions, data collection, data analysis, and interpretation of findings in this version of the dissertation text?

Explanation of the recommendation: In the section “General research observations” of this version of the dissertation text, it is stated: “The main objective of this study is to build an approach to national security as a product of sustainable peace and security achieved through genuine reconciliation (see Chapter 1, point 1.2). This research, presented and explained in detail in this chapter, provides five important answers or conclusions related to the main objective of this study: a) it provides an answer to the question of what kind of security people aspire;…” However, there are no direct answers to the research questions posed:

1. “How has the lack of trust and mutual respect affected the security doctrine of both sides?
2. “How has the failure to reconcile affected both sides’ overall stability?
3. “How does the lack of trust affect the perception of the personal and collective security of ordinary people?”
7.2 On conclusions
I recommend that the author of the doctoral dissertation consider (and answer to themselves) the following questions regarding the conclusions:

1. Are the conclusions succinctly and clearly formulated?
2. Are the conclusions logically derived from the evidence obtained through this research?
3. Are the conclusions adequately linked to the research objectives?
4. Are the conclusions adequately connected to the research questions posed?

Is the assertion “This study has shown that, despite the declared strategic goal of EU integrations, instead of pursuing a policy of achieving security through cooperation, both sides have decided to resort to offensive/defensive paradigms.” honestly an answer to the research question “How has the lack of trust and mutual respect affected the security doctrine of both sides”?
Justification for the recommendation: The section “General research observations” of this version of the dissertation states: “In chapter 1, we posed the question ‘How has the lack of trust and mutual respect affected the security doctrine of both sides?’. This study has shown that, despite the declared strategic goal of EU integrations, instead of pursuing a policy of achieving security through cooperation, both sides have decided to resort to offensive/defensive paradigms. Despite its simple construct, this conclusion is essential.” (p.173)

Concluding assessment:

The following subset of identified weaknesses and shortcomings in this version of the doctoral dissertation text: the absence of operational definitions of key variables contained within the formulations of research questions; the lack of information on elements of “nonresponse error”; inadequate alignment between the research questions, data collection, and data analysis; the absence of a description of how the “framework analysis” was conducted; insufficient clarity in connecting conclusions with the research questions and research objectives, constitute – in my opinion – sufficient grounds for the assessment that, from a methodological perspective, this version of the dissertation text requires a series of significant improvements (which relate to most of the questions recommended to the author of the dissertation for consideration).

References:

1.How to write a PhD Thesis, available at https://www.researchgate.net/publication/366781144_HOW_TO_WRITE_YOUR_Phd_THESIS_THE_EASY_HANDBOOK [Accessed, 16 July 2024]
2.Institute of Education, Criteria for assessing PhD thesis, available at http://www.robertfeldt.net/advice/univ_victoria_phd_criteria.pdf [Accessed, 16 July 2024]
3.QAA, Characteristics Statement, Doctoral Degree, available at https://www.qaa.ac.uk/docs/qaa/quality-code/doctoral-degree-characteristics-statement-2020.pdf [Accessed, 16 July 2024]
4.A Guide for Internal and External PhD Examiners, The University of Manchester, https://pure.manchester.ac.uk/ws/portalfiles/portal/31671759/FULL_TEXT.PDF [Accessed, 16 July 2024]
5.Guidelines for Examiners of Doctoral Degrees, University of Bath, available at https://www.bath.ac.uk/publications/guidelines-for-research-examiners/attachments/guidelines-for-examiners-of-doctoral-degrees-july-2021-plus-covid-note.pdf [Accessed, 16 July 2024]
6.A Practical Guide to Dissertation and Thesis Writing, available at https://www.cambridgescholars.com/resources/pdfs/978-1-5275-3681-4-sample.pdf [Accessed, 16 July 2024]